• View
• Edit
• History
• Attach
• Print
• Backlinks

ConvertTABLE

Question

How can I make it easier to convert HTML pages with tables full of data into advanced tables?

Answer

I had this problem with AzRepeaters.Net. I needed to convert about 20 pages filled with tables of data into a PmWiki's advanced table markup

Currently, my solution is a Linux one using the Bash command line and not a PmWiki cookbook recipe.

So, I ran this command line on a bash console on the HTML pages I needed to convert....

Let me explain what is going on with this command line.

First, be very careful with the syntax. The symbol you see above " | " is the key on your keyboard just about the enter key. It creates a pipe that streams data from one command to another until you get a final output. In other words, it is a necessary part of the command string.

I start the stream by using cat filecontainingtabledata.html. I am doing this to illustrate that it is streaming the contents of an HTML page into a series of commands.

After creating the data stream, I then pipe it into sed to delete some empty lines and pipe into sed again to delete empty trailing spaces ( sed '/^$/d' | sed 's/[ \t]*$//' ).

Then it is piped into tr where I delete all the newlines and carriage returns ( tr -d [:cntrl:] ). After that, it is piped into tr again to squeeze out repeating blanks ( tr -s [:blank:] ).

Then I pipe it into sed to get every < to begin a new line. This was the part that Patrick thought was clever. Anyway, it is done with sed 's/</\n</g'.

The reason it might be considered clever is that sed works on one line at a time. Even though a lot of data is being shipped thru the pipe, it still is parsed one line at a time.

This makes it easier to do line be line editing with a streaming editor like sed. And, as you will see later, I use it to do little but important tricks as the data streams thru each pipe.

Then I pipe it again into sed and convert every <tr> into (:cellnr:) and pipe it into sed again and covert every <td> into (:cell:) with sed 's#^<[Tt][Rr].*>#(:cellnr:)#g'=] and then sed 's#^<[Tt][Dd].*>#(:cell:)#g' .

O.K., now that is done, we can pipe it again into sed to remove all the HTML tags with sed -e :a -e 's/<[^>]*>//g;/</N;//ba'.

Then, I want to run everything together as one line. You may see why in a moment. I continue piping and use tr -d [:cntrl:] to do this step.

Now that everything is together. I can pipe it into the next step and set up each occurence of (:cell to start a new line. I do that with sed 's/(:cell/\n(:cell/g'.

Now, I finally pipe it for the last time where I just grep for for every line that starts with (:cell and use > to redirect the output into a file called data_in_pmwiki_markup.txt.

I then edited it in a text editor. I know that Patrick talked about making this into a another PmWiki recipe since most of the cookbook is that. But this will give you a start until that is done.

Neat Little Add On to make it work better

By the way, in case you do run this script, you will find that you create an extra (:cell:) just after the (:cellnr:). The way to get rid of that extra (:cell:) is to add this to the script by piping the stream into the commands tr -d [:cntrl:] | sed 's/(:cellnr:) (:cell:)/(:cellnr:)/g' | sed 's/(:cell/\n(:cell/g'.

You will have to adjust the spacing of the blanks in sed 's/(:cellnr:) (:cell:)/(:cellnr:)/g' to get it to delete the extra (:cell:).

Caveat and work around!

My script does remove HTML tags that you might want to include in the final data. I had that problem too. I adjusted the script with a sed command that took advantage of the fact that every tag has its own line.

This means you can change tags like <A HREF=http://www.somedomain.com to [[http://www.somedomain.com. Then on the next to final command, you can change the final bit of the tag left over of > to ]] .

I had to do this also because AzRepeater.Net has links showing where repeaters are on a map.

Notes and Comments

''The newest the latest. And please don't forget to end by date and name. Thanks)

Versions

date of publication : 2005-09-02 : name of the cookbook - version 00007

• This recipe was last tested on PmWiki version:
• This recipe requires at least PmWiki version: and (any other recipes)

See Also

• ConvertHTML - Convert an HTML page to PmWiki markup

Contributors

Brooks Kelley 17 September 2005

Comments

See discussion at ConvertTABLE-Talk