Saturday, July 2, 2011

Woohoo, 3-way!

I added my third router and machine, but couldn't get multicast to go more than one hop... it turns out that when I tried to test msdp and pim without tunneling them, msdp worked, pim didn't, so only pim got changed back to the tunneled address. Changing msdp to the tunnelled address made it work almost immediately!

Config dump

interfaces {
    em0 {
        unit 0 {
            family inet {
                address 10.1.1.198/8;
            }
        }
    }
    em1 {
        unit 0 {
            family inet {
                address 192.168.11.1/24;
            }
        }
    }
    em2 {
        unit 0 {
            family inet {
                address 192.168.2.1/24;
            }
        }
    }
    em3 {
        unit 0;
    }
    gre {
        unit 0 {
            tunnel {
                source 192.168.11.1;
                destination 192.168.11.2;
            }
            family inet {
                address 192.168.101.1/30;
            }
            family inet6 {
                address 2001:4428:251:2::1:1/120;
            }
        }
    }
    ipip {
        unit 0 {
            tunnel {
                source 192.168.2.1;
                destination 192.168.2.2;
            }
            family inet {
                address 192.168.201.1/30;
            }
            family inet6 {
                address 2001:4428:251:2::1/120;
            }
        }
        unit 1 {
            tunnel {
                source 192.168.2.1;
                destination 192.168.2.3;
            }
            family inet {
                address 192.168.202.1/30;
            }
        }
    }
    lo0 {
        unit 0 {
            family inet {
                address 1.1.1.1/32;
            }
        }
    }
}
routing-options {
    interface-routes {
        rib-group inet if-rib;
    }
    rib-groups {
        multicast-rpf-rib {
            export-rib inet.2;
            import-rib inet.2;
        }
        if-rib {
            import-rib [ inet.2 inet.0 ];
        }
    }
    autonomous-system 65000;
}
protocols {
    igmp {
        interface all {
            version 3;
        }
    }
    bgp {
        local-as 65000;
        group branch1 {
            type external;
            export [ to-branch1 allow-all ];
            peer-as 65001;
            neighbor 192.168.201.2 {
                family inet {
                    any;
                }
            }
            neighbor 2001:4428:251:2::2 {
                family inet6 {
                    any;
                }
            }
        }
        group branch2 {
            type external;
            export [ to-branch1 allow-all ];
            peer-as 65002;
            neighbor 192.168.202.2 {
                family inet {
                    any;
                }
            }
        }
    }
    msdp {
        rib-group inet multicast-rpf-rib;
        export allow-all;
        import allow-all;
        group test {
            peer 192.168.202.2 {
                local-address 192.168.202.1;
            }
            peer 192.168.201.2 {
                local-address 192.168.201.1;
            }
        }
    }
    pim {
        rib-group inet multicast-rpf-rib;
        rp {
            local {
                address 192.168.101.1;
                group-ranges {
                    224.0.0.0/4;
                }
            }
        }
        interface all {
            mode sparse;
            version 2;
        }
        dr-election-on-p2p;
    }
    rip {
        group gateway {
            export gateway-rip;
            neighbor em0.0;
        }
    }
}
policy-options {
    policy-statement allow-all {
        then accept;
    }
    policy-statement gateway-rip {
        from protocol [ direct bgp ];
        then accept;
    }
    policy-statement reject-all {
        from protocol rip;
        then reject;
    }
    policy-statement to-branch {
        from protocol [ direct local ospf bgp static rip pim ];
        then accept;
    }
    policy-statement to-branch1 {
        from protocol [ direct local ospf bgp static rip pim ];
        then accept;
    }
}

As you can see, I've started setting up IPv6 addresses on the routers. I've got RA and stateful DHCPv6 working on my real network, so there's no point muddying up the config here. By the way, it turns out you can have as many tunnels as you like - turns out stacking routed gre/ipip interfaces is totally okay. I hope to have some IPv6 multicast results this evening, so stay tuned

Clockwork Olive: multicast update

After much pissing around it turns out multicast does work, but emcast has been having problems. Dbeacon runs well in super verbose mode, emcast receives the info, but just doesn't seem to send very well - it could be that the olives are just being shit and dropping packets though.

Want to see the config?


interfaces {
    em0 {
        unit 0 {
            family inet {
                address 192.168.2.2/24;
            }
        }
    }
    em1 {
        unit 0 {
            family inet {
                address 192.168.12.1/24;
            }
        }
    }
    gre {
        unit 0 {
            tunnel {
                source 192.168.12.1;
                destination 192.168.12.2;
            }
            family inet {
                address 192.168.102.1/30;
            }
        }
    }
    ipip {
        unit 0 {
            tunnel {
                source 192.168.2.2;
                destination 192.168.2.1;
            }
            family inet {
                address 192.168.201.2/30;
            }
        }
    }
    lo0 {
        unit 0 {
            family inet {
                address 1.1.1.2/32;
            }
        }
    }
}
routing-options {
    interface-routes {
        rib-group inet if-rib;
    }
    rib-groups {
        multicast-rpf-rib {
            export-rib inet.2;
            import-rib inet.2;
        }
        if-rib {
            import-rib [ inet.2 inet.0 ];
        }
    }
    autonomous-system 65001;
}
protocols {
    igmp {
        interface all {
            version 3;
        }
    }
    bgp {
        local-as 65001;
        group olive {
            type external;
            family inet {
                any;
            }
            export to-branch1;
            peer-as 65000;
            neighbor 192.168.201.1;
        }
    }
    msdp {
        rib-group inet multicast-rpf-rib;
        group test {
            peer 192.168.201.1 {
                local-address 192.168.201.2;
            }
        }
    }
    pim {
        rib-group inet multicast-rpf-rib;
        rp {
            local {
                address 192.168.102.1;
                group-ranges {
                    224.0.0.0/4;
                }
            }
        }
        interface all {
            mode sparse;
            version 2;
        }
        dr-election-on-p2p;
    }
}
policy-options {
    policy-statement allow-all {
        then accept;
    }
    policy-statement to-branch1 {
        from protocol [ direct local ospf bgp pim ];
        then accept;
    }
}


I'm going to be a bastard any sources except this one. I'm tempted to chalk the emcast send failure down to packets simply being dropped, and maybe try a test VLC stream if I can be bothered with that, but this was only meant to be a means to an end - the next step is IPv6 multicast!

Thursday, June 30, 2011

JunOS Router testbed part 3: multicast still not working

So after a bit of research and tons of failed attempts, I've discovered that the olives really don't like multicast. Some people have had used a patch to enable it for OSPF on earlier versions of JunOS, but there's nothing for later versions (since these run fine), although MSDP and PIM still don't work.

I had heard about people using gre tunnels, and can confirm that this works. Olives only let you have one of each type of tunnel (due to there being no PIC's installed) so I used an ipip tunnel to connect two routers, got PIM and MSDP working, then gre tunnels to my two ubuntu boxes (as per http://knol.google.com/k/juniper-hacks/gre-tunnel-between-a-linux-host-and/1xqkuq3r2h459/43#).

I can see the routes filling up the MSDP table, dbeacon seems to sense get some sort of communication, but it still looks like multicast traffic isn't being routed properly.... at least it's getting across all the links now

Friday, June 17, 2011

JunOS Router testbed part 2

My topology has since become quite complicated, so I thought it would be best to draw a picture:
The fourth olive (meant to branch off like olive2 and olive3 with a separate AS number, tap interface and Ubuntu virtual machine) has been left out for simplicity at this stage. The main problem with my original design was that layer 3 separation wasn't enough - multicast skips routers at layer 2 - so I needed to give each box its own tap interface. To go with the diagram, here's the config from olive1 and olive2 (olive3 is basically the same as olive2 - this is an exercise for the reader)

Olive 1:


interfaces {
    em0 {
        unit 0 {
            family inet {
                address 192.168.2.1/24;
                address 192.168.11.1/24;
                address 10.1.1.198/8;
            }
        }
    }
    lo0 {
        unit 0 {
            family inet {
                address 1.1.1.1/32;
            }
        }
    }
}
routing-options {
    autonomous-system 65000;
}
protocols {
    bgp {
        local-as 65000;
        group branch1 {
            type external;
            export to-branch1;
            peer-as 65001;
            neighbor 192.168.2.2;
        }
        group branch2 {
            type external;
            export to-branch;
            peer-as 65002;
            neighbor 192.168.2.3;
        }
        group branch3 {
            type external;
            export to-branch;
            peer-as 65003;
            neighbor 192.168.2.4;
        }
    }
    rip {
        group gateway {
            export gateway-rip;
            neighbor em0.0;
        }
    }
}
policy-options {
    policy-statement gateway-rip {
        from protocol [ direct bgp ];
        then accept;
    }
    policy-statement to-branch {
        from protocol [ direct local ospf bgp static rip ];
        then accept;
    }
}
Olive 2:

interfaces {
    em0 {
        unit 0 {
            family inet {
                address 192.168.2.2/24;
            }
        }
    }
    em1 {
        unit 0 {
            family inet {
                address 192.168.12.1/24;
            }
        }
    }
    lo0 {
        unit 0 {
            family inet {
                address 1.1.1.2/32;
            }
        }
    }
}
routing-options {
    autonomous-system 65001;
}
protocols {
    bgp {
        local-as 65001;
        group olive {
            type external;
            export to-branch1;
            peer-as 65000;
            neighbor 192.168.2.1;
        }
    }
}
policy-options {
    policy-statement to-branch1 {
        from protocol [ direct local ospf bgp ];
        then accept;
    }
}

And here's a show route from olive 2

inet.0: 12 destinations, 13 routes (12 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[BGP/170] 00:31:58, MED 3, localpref 100
                      AS path: 65000 I
                    > to 192.168.2.1 via em0.0
1.1.1.1/32         *[BGP/170] 00:31:58, localpref 100
                      AS path: 65000 I
                    > to 192.168.2.1 via em0.0
1.1.1.2/32         *[Direct/0] 00:32:02
                    > via lo0.0
1.1.1.3/32         *[BGP/170] 00:25:00, localpref 100, from 192.168.2.1
                      AS path: 65000 65002 I
                    > to 192.168.2.3 via em0.0
10.0.0.0/8         *[BGP/170] 00:31:58, localpref 100
                      AS path: 65000 I
                    > to 192.168.2.1 via em0.0
192.168.2.0/24     *[Direct/0] 00:32:02
                    > via em0.0
                    [BGP/170] 00:31:58, localpref 100
                      AS path: 65000 I
                    > to 192.168.2.1 via em0.0
192.168.2.2/32     *[Local/0] 00:32:02
                      Local via em0.0
192.168.11.0/24    *[BGP/170] 00:31:58, localpref 100
                      AS path: 65000 I
                    > to 192.168.2.1 via em0.0
192.168.12.0/24    *[Direct/0] 00:31:09
                    > via em1.0
192.168.12.1/32    *[Local/0] 00:31:09
                      Local via em1.0
192.168.13.0/24    *[BGP/170] 00:25:00, localpref 100, from 192.168.2.1
                      AS path: 65000 65002 I
                    > to 192.168.2.3 via em0.0
218.101.61.124/32  *[BGP/170] 00:31:58, MED 2, localpref 100
                      AS path: 65000 I
                    > to 192.168.2.1 via em0.0

It's all going well so far - putting each subnet on a different tap interface stops them cheating and using layer 2 for multicast, so now I can start getting PIM-SM set up (IPv4 only for starters)

Router testbed with JunOS olive on Virtualbox

I did an SRX course earlier in the week and we got to use Olive virtual machines to play with what we had learned. I'd tried making my own but got into trouble when actually installing the package, so I took a copy of this olive (8.3) and tried to get it to work at home. The first results were less than ideal - they would run fine without crashing, but setting addresses had to be done on the commandline with ifconfig rather than in the interfaces stanza. Not only this, but routing was totally broken - not even OSPF would work!

I had read that JunOS 9 didn't suffer from this, and tonight I acquired a copy of JunOS 9.6. The upgrade went smoothly (needed a force as the leftover diskspace wasn't enough, but it installed fine) and it automatically picked up the addresses from the interfaces stanza. OSPF worked fine between 4 of them, so the next thing was to use BGP to set up a basic layer 3 topology with 3 routers all with a single peering with the router in the middle.

If you've done JunOS BGP before then you'll know this is trivial - I made my life easier by making the export policy take routes from direct, local and bgp (which means readvertising happens automatically). The point of this testbed was simply to check my connectivity.

It did all work in the end, and now I'm on to part two - testing out multicast. The plan is to get a couple of virtual interfaces on a real machine, set up multicast between the routers, and have each virtual interface on a subnet owned by a different router. They're all connected to the same bridged interface which means the layer 2 topology has everything effectively hanging off the same switch, so this will be successful if I can get multicasts happening between the different subnets. This is somewhat trivial though, and the next step is to get IPv6 connectivity and testbed IPv6 multicast - if it works, then I'll put up some detailed instructions of all the ins and outs!

Sunday, May 15, 2011

Tuntap - going down the rabbit hole

Firstly I'd like to begin by apologising if the title makes this sound at all interesting - it drove me completely fucking crazy for most of last week. The issue I had was a packet capture probe which would stream the data elsewhere, but I didn't have the luxury of being able to cache the gigabytes of data it was putting out every minute. I tried using netflow with ntop (since I'd had absolutely no experience with flow analysers) and this gave me a good start, but there was pretty much nothing I could do to manipulate the data apart from a few hall of fame style charts.

The next step was to try snort and set up some rules based on ip ranges (a step ahead of ntop in netflow mode) and then run it through snortalog to make it a bit easier to view, but snort doesn't take data directly from a single port. I tried to dump it from netcat into a named pipe, but snort doesn't read "special" files... The next option was to start playing around with tap interfaces.

The tap and tun interfaces are virtual NICs which you can apparently send data directly to. I had no luck getting this to work - my tap0 didn't give me a /dev/tap0 file to pipe to, so I ended up back at square one... almost. The final key was using tcpreplay to replay from the named pipe to tap0, and then attaching snort to it. It ended up working, but being asynchronous, snort ended up missing half the packets since the system was busy trying to pipe data to this virtual interface that nobody else could read from...

And in the end? We all lived happily ever after. I found the tcpdump filter I wanted, set up tshark to read from the named pipe, and it's all working, all thanks to the almost unusable tuntap interfaces.

Monday, May 9, 2011

Two part network authentication

We've just changed to using two part authentication for google at work, and it seems to do the job well - unintrusive, unless you lose your phone. The idea is simple - retain your current password, but use a second one-time password which is generated by an app on your phone. When you first set it up, you paste a massive password into your phone if you're unlucky enough to not have an iPhone 3GS or higher (with a camera capable of reading the barcode on the screen). This acts as the seed for a random number generator, which is combined with the current time to the nearest minute to generate one-time passwords that can only be predicted if both sides have their time synchronised and share the same password.

This is cool in itself, but after a conversation with one of my colleagues I thought it would be cool to extend it, and combine it with the concept of port knocking.

Port knocking, for those who don't know, is at worst another layer of security through obscurity, but at best is another channel for confirming knowledge of shared secrets. Normal firewalls try to make it difficult for potential hackers by detecting when they scan for open ports (which correspond to network services) and then not confirming or denying whether or not any of the ports are open. Port knocking goes further, by making all ports appear closed, unless the IP attempting to connect to a service has recently queried a list of ports in the correct order (a sort of secret knock if you like).

A traditional port knock is a predictable sequence, which can be easily inspected by routers along the way. To add a further layer of security, setting the TCP sequence number to the value of a hash of the packet combined with a shared secret - thus ensuring a port knock from one IP can't be replayed later from another.

But what if we want to make the sequence itself unpredictable? If we restrict ourself to just 256 ports, and make our port knock sequence 16 ports long, then we can convert the output from a cryptographic hash into a sequence of ports to query. Sharing a secret in advance, and salting this with the current time to the nearest minute allows us to create per-session portknocks. And the icing on the cake? Add the IP to the time as a second salt, allowing the client to perform the portknock in plain sight, and then be allowed access to a totally hidden port.

OR..... we could just use IPSEC AH

Saturday, April 9, 2011

Server upgrade: day 2

After deleting a few packages that had been installed from the ubuntu stream and were causing major issues, the upgrade somewhat took care of itself. TFTPD-HPA isn't incredibly happy at the moment, but I'll sort that out next time I feel the need to PXE boot anything (there may be a tutorial on exactly how I got BartPE to PXE boot, which would be just a mix of all the information that is already floating around on the internets).


I've already got a working config for the WIDE DHCPv6, so migrating to the ISC version wasn't going to be too difficult - fortunately I found a sweet tutorial which made life easier. Once I'd got my new config set up, all that needed to be changed was /etc/init.d/dhcp with a couple of extra lines



                echo -n "Starting DHCPv6 server: "
                start-stop-daemon --start --pidfile $DHCPD6PID \
                        --exec /usr/sbin/dhcpd -- -q -6 -cf /etc/dhcp/dhcpd6.conf $INTERFACES
                sleep 2


                if [ -f "$DHCPD6PID" ] && ps h `cat "$DHCPD6PID"` >/dev/null; then
                        echo "dhcpd6."
                else
                        echo "dhcpd6 failed to start - check syslog for diagnostics."
                fi


It hasn't crashed yet, and I'm hoping that it will be a bit more resilient than the WIDE package. I'll let you know how my testing goes with all my different operating systems here!

Friday, April 8, 2011

Nagios, DHCPv6, and migrating Debian servers

Migrating debian


I convinced myself I'd keep this updated after I started work, but this was assuming that I'd still have time to play with my server alongside having a real job and trying to maintain some sort of social life. I did however end up a little bit tipsy last Thursday, which prompted me to start migrating my server from it's original 8GB home to the 250GB drive that has been there for the last two years under the name of /usr/bigdisk for obvious reasons.


My first hurdle was underestimating how long it takes to copy 8GB over IDE on an old pentium 4 (on a side note, describing a pentium 4 as old still makes me feel like a bit of a dinosaur given i vividly remember my first 586) so after having unscrewed my case and plugged in the CD-ROM without electrocuting myself, and booting up backtrack (version 2, because 4 is on a dvd, and 3 was a bit shit), I started what I thought was a sensible cp /mnt/sda1 /mnt/sdc2


Fast forward an hour and a half, and ignoring the part where I converted the original partition to ext2 (by deleting the journal) and shrunk it which took about half an hour by itself (and is already adequately documented everywhere else on the internet) and created the new partition (total of 10 seconds work), I got to the stage where I installed grub (after a chroot to /mnt/sdc2 because bt2 only has LILO) and was ready for a reboot, only to find that half of my stuff wouldn't work because EVERYTHING was 755 to root.


This was about the time I decided it was good to go to bed, since I had to get up for work in less than five hours. The next day I did the procedure all over again, except with the help of the -a switch to the cp command (keeps permissions and timestamps the same - don't leave home without it), made sure to set up fstab, set the partitions back to ext3, and set up grub to use the correct partitions. Finally, everything booted, and I was good to go.


Why did I migrate? The drive was only 95% full, and I thought maybe that was why my wide-dhcpv6-server was crashing... but this wasn't the case in the end.


DHCPv6
I'd not had any problems with wide-dhcpv6-server until recently, so I figured a nearly-full hard drive was why it seemed to crash for no reason every second day. This wasn't the case however, and my research showed that the WIDE DHCPv6 project stopped about three years ago. It turns out that the official ISC DHCP server incorporates both now (although they need to run in separate instances), but to install this, I needed to upgrade to squeeze, so once all 2.4GB of packages come down, I'll let you know how this goes for me.


Nagios
We use this at work to keep an eye on the network, and after writing a plugin to report on light levels on SFP's I decided it made sense to install it at home. This helped me track the DHCPv6 problem, but also made it easier to see the moment anything went wrong. It's only a simple setup at the moment, with periodic pings to google through my IPv4 and IPv6 gateways, and checks on the HTTP and DHCP servers, but already I feel like I have a way closer eye on the health of my server.

Saturday, February 26, 2011

PhotoMagnetic

This is a project I've been wanting to do for a while, and I finally got around to putting the code together yesterday.

It's based on a project called JSteg, the idea of which is to simply alter JPEG data after the lossy compression phase has been completed.

Microsoft has a fairly good explanation of how JPEG works and this can be simplified into three main phases:

  1. Downsampling
  2. Transforming and quantising
  3. Lossless compression
Downsampling
The first phase represents the image in terms of brightness and chrominance, brightness being measured on a one dimensional scale, and chrominance on a two dimensional scale with one axis being red-green, and the other being blue-yellow. Experiments have shown that the brightness channel is what effects our perception of images the most - the JSteg page has an example of how far the chrominance channels can be downsampled  before noticable changes start to appear.

Transforming and quantising
The data is then subjected to a discrete cosine transform which effectively replaces the pixel values with frequency values. These are then quantised (smoothing off the rough edges if you like) and then rearranged in order to place all the resultant 0 values next to each other. You can imagine what the next step is now.

Lossless encoding
The adjacent zeroes are perfect fodder for run length encoding, followed by huffman coding. The resultant space savings come not only from direct reductions in the space used by chrominance values, but also a simplification of the data such that traditional compression can be used.

Hiding a 160 bit hash
This is actually the easiest part. All the complicated work is done in the lossy stage, so all that needs to be done to hide our hash is to alter the file before it is losslessly compressed. The basic operation is as follows:
  • Decompression
  • Altering the coefficients
  • Compression
The way we hide the data is overwriting the LSB of the coefficients, which results in a minor change spread across that part of the image. The advantage of saving a fixed length of data is that no extra metadata needs to be stored, so that any file that hasn't had anything encoded will still produce a hash, it just won't have any meaning, so it gives the encoding some amount of deniability.

You can download the project from here if you'd like to try it out for yourself.

Monday, February 14, 2011

Ściągnij z iply

Kolega mi polecił programa "IPLA" abym obejrzeć seriałów polskich. Oczywiście, to mi się spodobało, ale nie lubię objerzeć online - wolę ściągnąć a potem obejrzeć, abym nie wyczerpać internet.

Więc, uradziłem napisać swojego program, przez co mogłbym tak zrobić. Na szczęście było prosto, bo IPLA używa XML, a mogłem tworzeć jakiegoś XSLT który mi pomógł. Wreszice mi się udało, ale trudno użyć UTF-8 z stdin'a z windowsem - nie mogłem po prostu użyć fgets() lub scanf() a raczej potrezebowałem ReadConsoleW(), SetConsoleCP() i SetConsoleOutputCP().

Program ściągnij z iply można pobrać, więc spróbuj, i mów mi co masz na myśli!

Thursday, February 3, 2011

ASFDump: I'm sick of it

The plugin is dead in the water. XPCOM and NSPR lack documentation and working samples, and I've already spent enough time getting this project up and running. The command line app works, but as for the plugin, somebody else will have to make that.

The project can be found here, feel free to let me know what you think of it

Wednesday, February 2, 2011

ASFRecorder part 3: Making the plugin

This, I told myself, is the easy part - it's all smooth sailing from here!

No such luck unfortunately, although I found comfort in the knowledge that making a plugin for firefox meant I could actually believe that the specification was being somewhat followed. This part was much easier in comparison to the ASF parsing, so I can't really complain too much.

The way binaries are included in addons in firefox is fairly straightforward, and there are good tutorials (my plugin was based more or less on this one with a few adjustments to the makefile) so after the issues were sorted out I had my test plugin ready to go.

With the test plugin sorted, the next step was to add functionality to detect the loading of .asf files. There are a number of different plugins that download .flv files, and a quick look through the (copyrighted) source of these turned up extensive use of observer notifications, and a check against the (GPL licensed and therefore stealable) Tamper Data source confirmed this for me. A look over some of the firefox articles showed me how to extract the URLs, which leaves me set for the next steps:


  • Add observers to the test plugin
  • Cache .asf (and .wmv) file requests
  • Maaaaaaaaaaybe look at response headers to check for MIME types I like
  • Load the ASFRecorder and post processing code into the binary part of the module
  • Maybe add some callback options to give an idea of progress
  • Package up and start distributing
It looks fairly easy from here, maybe the callback stuff will be a pain, but I'm hoping I can get it all sorted by the weekend

Tuesday, February 1, 2011

ASFRecorder part 2: fixing up streaming output

The project is coming along quite well, and I mean quite well, and I'm at a stage where I'm almost ready to build a plugin framework to host a beta. Here's a quick run through what I've had to do so far to get the output to a point where more than just VLC will play


  • Remove all references to the extra streams that are filtered out, including removing a few headers, rebuilding the header section in the process
  • Add padding lengths  to each data packet (because packet sizes are fixed, the server actually breaks its own standard by not setting the padding lengths, so although this is a trivial exercise, it was more work than I wanted to do)
  • Add packet length values to a sparse minority of data packets (something like one in 100,000, but it breaks avidemux) by manually summing the payload lengths, and then setting the padding lengths accordingly
  • Remove the null data packets (once again, these are fixed length, so easy to detect and skip, but this not only breaks the standard, I strip about 10% off the size of the file once these are removed)
Was it worth it? Turns out trying to appease avidemux wasn't the best goal, as its wmv reader module looks quite seriously broken (maybe it doesn't read the keyframes but the output has tons of artifacts), but now the output works on that, vlc, windows media player, and totem movie player.

It should all be smooth sailing from here though - just putting the post processing routines into a single module, getting ASFRecorder up to scratch, possibly tying the two modules together (or maybe that can be the second release), then doing all the GPL documentation and putting the demo plugin together.

I just hope I've gotten the hard part out of the way now ;)

Thursday, January 27, 2011

ASFRecorder

As fantastic as streaming audio and video are, sometimes it's nice to reply things later. As far as websites which host .flv go, the act of saving the media file is fairly trivial, and is done by a wide range of browser plugins. Websites which actually rely on a client-side stream player are a different story though...

I was after a simple option, and ideally something single-purpose. Apps like VLC are up to the task, but a bit too big to bundle in a plugin, and lack some desired features (like resuming, in the case of VLC). There is a Japanese app called GetASFStream which appears to do a good job, and even rebuilds the header based on the streams chosen (which as far as I can tell simply chooses the first audio and video streams). There's a couple of problems though - it's in japanese, doesn't appear to be open source, and doesn't appear to have commandline support. So the search continues...

I then stumbled across the code for an application called ASFRecorder. It appears to be unchanged since 2001, and as a result, lacks a few features.

For starters, it doesn't recognise the content type "application/vnd.ms.wms-hdr.asfv1", which Microsoft gives little data on, but it appears I'm not the only one to come across this problem.

Once I'd patched the code to take newer ContentType values, I came to another problem - it doesn't parse the payload parsing information section properly - instead of actually interpreting it as per the spec (which is freely downloadable), it appears as if the original author picked a few common values, or found them from trial and error, instead of calculating the length as a general case function. Once this is fixed, ASFRecorder happily dumps the stream to a file, but we're not done yet!

As described here, ASFRecorder only downloads the first stream, and dumps the header has it. This is quite messy, but the attempt is quite a valid one - if the stream is dumped directly to a file, then in theory it can be played straight back. However, it appears the code was only ever written for single stream broadcasts, which means the user is limited to either audio streams, or soundless video streams (ie. radio or porn), and so despite the file being written with a multi-stream header, only the first stream is saved.

I decided to undertake the rewarding task of building my own header parser app (which I might post if anybody cares enough to look at the code) so I could compare the (incorrect) output of ASFRecorder with the (correct) output of GetASFStream. After comparing the two headers, and trying (unsuccessfully) to simply paste the header from the correct dump onto the incorrect one (which stopped it from playing completely), I discovered that the reason ASFRecorder was only dumping a single stream was because that's all it was asking for.

By changing the hardcoded reply message so that it asks for two streams instead of one, my dump is suddenly working correctly! However, there is a bit more tidying that needs to be done, and some of it is necessary before release, but there is also some I can save until later

  • Actually coding a general parser for the payload parsing information
  • Changing the handler to take all MIME types for ASF files
  • Parsing the return value from the server to give a proper selection of streams
  • Processing the whole header, writing a new one, and arranging the streams accordingly
  • Building a proper resume function to compensate for the new header
  • Somewhere along the line incorporating this into a browser plugin
And that's all.... looks fairly straightforward.

One final comment to make about the ASF file format - it's actually logical. The last time I messed with m$ file formats was DIB and WAV files, each with their own quirks and rude deviations from the standard. It seems when m$ does their writing from scratch, they do a proper job of it.