The method of creating Trap Books was devised by the D'ni people and involved subtly altering key lines of text of a linking book, not altering itself but rather partially severing its connection. The changes needed are so slight that anyone unfamiliar would overlook them. The Gateway Image would look proper, however anyone who would use such book, would be permanently trapped in the dark void of the Link. However if someone else use the Book, they would switch places, thus freeing the former prisoner (but at the cost of trapping oneself).
Atrus discovered that technique during a hunting expedition through the ruins of D'ni City with Gehn. He found a formula but as he feared that Gehn would confiscate once more his discoveries, he decided to leave the document behind. Years later, however, wishing to find a way to safeguard the Myst Library from greedy explorers, he managed to recall most of the formula, and recovered it with little experimentation.
When The Stranger linked to the D'ni-looking Trap Book, he described it as linking into a pure nothingness; he found himself locked in a black void accompanied only by fevered thoughts, and a feeling of loss and abandonment; the worst fate imaginable.
According to the Myst scenario, the Red and Blue Books are Trap Books, and after Atrus left for D'ni, Sirrus and Achenar had eventually been trapped in the inter-Age void, while looking for more Ages to rule. They sought for the Stranger to find the 5 pages in order to link to the books, thus freeing the brothers, while s/he'd be trapped there. This is seen in the two of the losing scenarios.
Some fan theories tried to reconcile these: At the end of the Myst game, it is believed that the brothers were indeed trapped in the void, but Atrus before burning the trap books, rewrote them back into into normal linking books, and the brothers each finally linked to their respective prison Ages (i.e. ages with no linking books in them). A fan theory is that Atrus didn't need to rewrite the prison books, but rather the destruction of the books forced Sirrus and Achenar into their respective prison Ages.
In Riven, Atrus' journal makes an analysis of what Trap Books are, and an essential puzzle for trapping Gehn, includes this exclusive property of Trap Books, getting trapped while switching places with another. Finding a way to incapacitate Gehn, Atrus tampers with a linking book to K'veer to lure him there and trap him.
In the game, Gehn, although he presumably knows nothing about the Trap Books, is skeptical, and forces the Stranger to use the book link to D'ni to check if the book is safe. The Stranger is indeed trapped there, but Gehn eventually can't resist and decides to link there himelf, trapping himself and freeing the Stranger.
The trap book used to capture Gehn was presumably slightly different to the two encountered in Myst, perhaps because the nature of the Linking panel and what it shows was retconned between the making of the two games. In the game, the panel continued to show a normal linking image even after it held a person. Strangely, near the end of the game, Catherine examines the book and somehow discerns that Gehn is trapped there, although it is not obvious how she can tell this.
These events are explained by Cyan Worlds as artistic license, as well as the communication through the Gateway Image. However, the switching of places between Gehn and the Stranger in Riven can not be explained canonically as no other books have this property, and there is no canonical explanation of how the Stranger trapped Gehn.
According to the fan theory that by "repairing" the trap book sends the prisoner to a proper Prison Age, Atrus could not have done it, because that particular book was made from a D'ni linking book; repairing the link would release Gehn there. Under this theory, Gehn should remain in the void indefinitely.
After the production of Riven, Cyan Worlds announced (by an e-mail written by Richard A. Watson, who was the first official mouth to dismiss what was established by the games) that Trap Books are not canonical and the Art did not have the power to write them. Instead, Watson revealed that Trap Books as seen in Myst and Riven, are historically Prison Books; however the Prison Ages could not be implemented in the game, as seeing them in the losing scenarios would be like "rewarding" the player with more explorations, instead of ending the game. Trap books were just a device to help the gameplay.