After two days at home I was somewhat crazed. I sent my mother to the bus station, went to Slaveykov for a while (where I found the other book by Georgi Markov on the same topic and a new one by Michael Crichton), after which I went to the anime meeting to see the people there (who were a bit less than usual, but we again had a hard time fitting in Bohemi).
След два дни стоене вкъщи бях се побъркал. Изпратих майка ми на гарата, постоях малко на Славейков (където си намерих следващата книга от “Задочните репортажи” и една нова на Майкъл Крайтън), след което отидох на аниме срещата да се видя и с хората там (които бяха в леко намален състав, но пак едвам се събрахме в Бохеми).
I’m almost OK, I just a bit snotty now…
I finished reading “Reports from Bulgaria” by Georgi Markov (maybe known also as “The truth that killed”), the books is really good and I recommend it to everyone that’s interested in the 1944-1969 period and the totalitarian period at all. The man combines the engineer and writer in a great way, and the result is really easy to read and full with information.
Почти се оправих, сега само съм малко сополив.
Изчетох “Задочни репортажи за България” на Георги Марков, книгата се оказа наистина много добра, и я препоръчвам на всички, които се интересуват от периода 1944-1969 и изобщо тоталитарния период. Човекът по някакъв много добър начин съчетава инженер и писател, и крайният резултат се получава не само лесен за четене, но и наситен с много информация.
Shortly – I had 38 deg. temperature through the day, I’m now reading “Zadochni reportazhi ot Bulgaria” (“Reports from Bulgaria”) by Georgi Markov, and I’m waiting to get better.
Накратко – имах 38 градуса температура през деня, чета си “Задочни репортажи от България” на Георги Марков и чакам да ми мине.
Happy birthday, Penchev, I wish you to be less absent-minded in the future :)
Честит рожден ден, Пенчев, пожелавам ти с възрастта да си все по-малко отнесен :)
За заинтересуваните – български mirror на лекциите от WhatTheHack има на http://phoebe.ludost.net/rawtapes/. Самата програма на лекциите може да се намери на http://wiki.whatthehack.org/, харесвайте си и теглете, надявам се bandwidth-а да ви се види достатъчен…
The day passed in the audit, I wrote my report, what’s left is tomorrow for chervarium to check his stuff and to finish it. We’ll see after it how are we going to implement the needed changes, it will probably be in Saturday evening.
Then I went with the CEO to eat at “Divaka” (he really liked the cooking :) ), I reminded myself of the great paunch soup there, we drank a bottle of Cabernet Gorchivka, and I went home on foot.
Деня мина в audit, написах си доклада, остава утре червото да прегледа неговите работи и да допълним. Ще видим как ще се планират нужните промени по системата, сигурно ще са в събота вечер.
После отидох с CEO-то да хапнем в “Дивака” (хареса му как готвят :) ), аз си припомних каква хубава шкембе чорба правят там, утрепахме една бутилка Каберне Горчивка, след което пак се разходих пеша до вкъщи. Хубаво нещо е да се раздвижиш :)
Today I was auditing an old project – I found some problems, tomorrow I’ll finish the check and will write a list of the needed changes.
In the evening I was at a Balkandji concert – I also took my mother with me, had a great time, got really tired, and intend to sleep for a long time…
Днес няколко часа audit-вах една система на стар проект – намериха се няколко проблема, утре ще довърша прегледа и ще напиша списък на нужните промени.
Вечерта бях на концерт на Балканджи – заведох и майка ми, раздвижих здраво куфелния мускул, уморих се като за трима, и смятам да спя дълго време…
I slept, ate, and slept again. In the evening I walked with my mother to get a package from the bus station, and after that we celebrated at home cheravium’s unfitness for military service.
Спах, ядох, и пак спах в общи линии. Вечерта се разходих с майка ми до гарата да вземем един колет, след което вкъщи празнувахме обявяването на червото за НГВС.
In the morning I got up at 5, to be able to get to the bus station at 6:30, and to go with Velin to Stara Zagora for Boyan’s wedding. We managed to find seats on the bus at 7, and got there in about 3 hours (the bus had a replicator of the climate, it was dripping inside like outside).
In the town we went to Happy, and ate pretty well, because we hadn’t had any breakfast. There we saw a tremendous rain, which made rivers flow, and at 12 we had some problems getting to the ritual hall, but in the end we arrived a bit wet and attended the ritual (which went well, there wasn’t anybody to shout “No, don’t do it!”). Then we went to a restaurant, where we drank, ate, the people danced, even I did it at Elvenpath by Nightwish (I really want to see myself at the recordings….).
Boyan, Marieta, congratulations :)
At some time Delian was going home, so me, Velin, Gogo and his girlfriend got into his car and took off for Sofia. The roads situation wasn’t really clear – the highway was closed and opened a few times before we took off, but we managed to travel safely. On the way we passed through Pazardzhik and made some photos of Marica – it had an incredible current, I’ll upload the videos somewhere… On the highway we got a flat tire and had to change, but we were home at dark. We went through a pizzaria to get some food, and went home.
On the way and in the pizzaria we had some really interesting talks on economics, which gave me a lot of food for thoughts…
Сутринта станах в 5, за да мога да се добера до автогарата в около 6:30, и да идем с Велин до Стара Загора за сватбата на Боян. Успяхме да си намерим места за автобуса в 7, и за около 3 часа се добрахме дотам (автобуса имаше репликатор на климата, и вътре капеше дъжд като отвън).
В самия град се занесохме до Happy, и хапнахме доста прилично, понеже не бяхме закусвали. Там ни хвана един невероятен дъжд, от който потекоха реки, и в 12 имахме малко проблеми да се доберем до обредния дом, но в крайна сметка малко мокри стигнахме и присъствахме на самия ритуал (който мина съвсем нормално, нямаше някой да вика “Недей бре!”). После отидохме в един ресторант, където пийнахме, хапнахме, хората танцуваха, даже и аз на Elvenpath на Nightwish (определено искам да се видя на записите…).
Бояне, Мариета, честито ви :)
По някое време Делян тръгна да се прибира и аз, Велин, Гого и приятелката му се натоварихме при него и тръгнахме към София. Не беше ясна ситуацията по пътищата, няколко пъти отваряха и затваряха магистралата преди да тръгнем, но успяхме да пътуваме нормално. По пътя минахме през Пазарджик и направихме малко снимки на Марица – беше едно невероятно течение, ще кача видео записите някъде… На магистралата едната гума се спука, та я сменяхме, но в крайна сметка по тъмно бяхме се прибрали. Минахме през една пицария да хапнем, и се прибрахме по домовете.
По пътя и в пицарията имахме много интересни икономически разговори, които пак ми дадоха много храна за размисъл…
Today at one of the projects we reinstalled a machine, I met my mother, and now I’ll be going to sleep, because tomorror in the early morning I’ll be going to Stara Zagora – Boyan is getting married…
Looks like I’ll rest when I die…
Днес в единия проект преинсталирахме една машина, видях се с майка ми, и сега мисля да спя, понеже утре в ранната сутрин пътувам за Стара Загора – Боян се жени…
Изглежда ще си почивам като умра…
It was incredible. Really indescribable.
(and I could just vilely stop here, with this excuse :) )
The event was mostly a gathering of ~3000 people at a camping ground in tents, where there were showers, normal toilets, food, drinks and 8-9 big tents (one of them could fit 1500 people inside) for lectures, bars, etc. There was a serious network (which I’ll describe belo, the schemes and some more detailed information can be found at wiki.whatthehack.org/index.php/FHQ). The core team wasn’t big, most of the work was done by volunteers, divided in teams, which was great, because not only everyone could participate, but a lot of work got done. I have no idea how this can be without the number of volunteers that were there (on this topic you might want to see Rop’s closing speech).
The event was closely pursued by an enormous quantities of rain, really enormous – it rained most of the time, the air was humid, dense, the tanks of the air conditioners in the NOC had to be drained every 8 hours, and there are even some photos of pumping the water under some of the tents. At 29/30 there was a big rain that made a lot of people to move to other places (especially the core team, who was at field K, which had no drainage). Some people slept in the NOC tent, I slept with the servers, and Walter had the experience to wake up and feel some electricity from the water on the floor in his tent. I wondered at some point if the people would change their bicycles for boats.
I wanted to describe all the people that I met, because they were really interesting, but it’s going to take too much space and time, and knowing my memory, there’s a great possibility to write something stupid, so I’ll just name them – Arien (core switches, full of ideas, Walter (head of the FHQ), Athilla (the main guy in the pulling the cables for the rings), Willem (tracking/monitoring switches in the field, Andre (field switches), Ernst (everything, this guy was unable to stay without something to do :) ), Reggie (also almost everything, always had a joke :) ), Maxim (head of the Wireless NOC, saved me one morning with a bottle of schnapps), Pim (who showed me how to configure a Juniper and didn’t scream at me for the things I did to the net :) ), Stephanie (words can’t describe her, when she smiled we had no need for the sun :) ), English Pete (he never stopped at one place, he was the adrenaline of the event :) ), Bassam Hassan (whom I couldn’t find at the event after his lecture to ask him a lot of stuff about Iraq, I’ll find his email from somewhere), and other known and unknown people…
I finally met Bine face to face – we’ve known each other from the time I started using Orkut, and had spoken about a lot of things, but seeing someone in real life is totally different. Too bad that both of us had too much work, and weren’t able to speak too much (she saved me from freezing by landing me her backup inflatable bed, for which I’m eternally grateful :) ).
What did a great impression on me was how many people are using Mac OSX – at least half of the latops were macs, and some people told me that I’ll switch to one someday (which I don’t feel believable, to be honest – I don’t like the OS, it’s not that nice to fiddle with :) ).
I’ll try to tell shortly about some fun moments in the event:
An optimist had ran arp spoof and wasa trying to make a ssh man-in-the-middle attack. He got caught in no time, and his photo showed up on the wiki :).
We filled the outside link at about 1/3 (looks like nobody had came to leech :) ).
A girl did a striptease in front of a webcam.
Someone had announced (jokingly) that the police officers that were at the event (they were 16) will be giving a lecture titled “Lawful interception”. A lot of people had gone there, the police guys were pretty surprised (here’s the entry in the Wiki, even AP fell for it :) )
In the end I was so tired and not really normal that I made a wireless cable and a wired/wireless converter (what they looked like I leave to your imagination :) ). That was mostly because a few days ago someone had managed to play a joke with an user, that he needs a wireless cable to use the wireless network… We gave as present one such cable to the wireless people.
In the last 2-3 days we used the NOC as a place to dry clothes – one of the ropes was the one that was used to pull the shutters, and the other one was an UTP cable, tied with cable ties.
When the police officers were going out on a round, the CERT car kept following them with it’s siren turned on. It was really amusing :)
With Athilla we were caught in the Megabit bar at some point and it turned out that there was only one free ethernet cable, so I connected my laptop and routed him through my wireless interface. It was a pretty fun exercise :)
Because of the weird laws there was no hard liquor there, but a lot of weed was smoked. Too bad I didn’t bring a bottle from Bulgaria, to make greyana rakiya… But Bobson managed to make a great dessert with vodka, dried cherries and chocolate, which we should test locally too. There were some schnapps-es, which were about 20% – which was good enough :)
One morning the water in the showers was pretty cold, and Pete joked in the Wiki that he did that on purpose, to better wake up the volunteers in the morning, to be ready for the work… I think that if he had really done that, we’d had him drowned in some ditch.
If you want more, the Wiki is a great source of information :).
The network setup was as follows:
The core network consisted of 3 Foundry BigIron switches (in the NOC, WhatTheBar and Megabit) connected with 10gbps links. The outside connection and all the inter-vlan routing were done on a Juniper M40, which also (at least for a while) was DHCP relay. In the field we used HP2626 and HP2824 switches, which with we had made two rings in a way that if power failed at some place it won’t be that big of a problem (the map is somewhere in the wiki :) ).
Logically we had some VLANs – one for the field (/19), one for the big tents (/19), one for the wireless users (/20), a few /24 for internal needs (administrative, switches, PXE for the wireless people), one for servers (/24), one for co-location (/23) and one for the OLSR people (/23 or /22).
For servers we had 5 machines for different services – pentium3 with small drives and a good quantity of memory. Two of them were HA cluster for some of the core services (DNS, DHCP, NTP), one was for Arien’s spongebob (a system for mitigating the arp traffic), one was for WINS server under Windows (which was done by Athilla), and the last one was for the switches’ administration (syslog, tftp, etc.). All of them were running debian except the windows machine, which I had configured, tuned, etc, they had ipv6 support and had almost no problems (there was some stuff around the dhcp, but it turned out to be because of the Juniper’s relay – too bad Pim wasn’t there, to do all possible tests). The only thing that troubled me was that all of the servers were front-mounted only, and I have no idea how did they stay there, they weren’t light.
(we could’ve used more power, then we could’ve hosted the wiki)
There was one machine from RIPE for snort, which had it’s own optical interface, and received mirror from the outgoing port of the juniper. The machine had a lot of problems, because FreeBSD 5.4-RELEASE couldn’t handle the load, and 6.0 died every 15 minutes, I have no idea what happened in the end.
(about this machine there was one really fun (for me) moment, one of the OpenBSD guys had came into the NOC tent and we spoke a bit on this problem, so he said that the polling of an interface was a really stupid idea (and that’s why it isn’t implemented in OpenBSD), and I asked him – ok, what if you have 100k interrupts per second? He said – yes, but what to do in the cases in which you don’t have so many, and I answered – you just switch between the two modes at different loads, like in Linux. We gave me an strange look and left, and I had this evil grin for half an hour.)
There was one machine brought by Pimto monitor the traffic on the interfaces – it was running only cacti.
The wireless people had 2 machines co-located – one for monitoring and one for dhcp and stuff, two big SUNs.
There were a few more SUN servers with a shared storage, but I never understood what it was used for.
We had some machines co-located by users – 3 in the NOC (one big HP that we used as a table, that had HP-UX running and was a shell server, one mac mini with OSX, which had an ircd connected to IRCNET, and AMD64 machine that was indexing the local FTPs, binary newsgroups, and had some vservers running (one of which was for the local OpenVPN server)), 2-3 in WhatTheBar (from archive.org, red 1U boxes).
All servers had separate IP addresses, and all services had addresses too, which made the moving between servers really easy, it also helped a lot with the heartbeat configuration.
We had full IPv6 support and routing.
The network had some fun moment, like PEG-DHCP. To the requesters we gave pegs with the last two octets of the ip address and a piece of paper with the rest of the configuration (IP prefix, DNS, etc.), which guaranteed that they have an unique address and weren’t dependable on the DHCP servers (which had it’s problems). The idea even has it’s own RFC… In the end because of the rain people were coming to get pegs to hang their wet clothes, and on the last day a guy came to get some of them to sell on ebay…
We had 15 km cable, from which we used about 3 km for the rings and for other stuff, the rest was used by the people for their connections in the field – we finished 11 km for one or two days. The principle was as follows – we had tools, jacks and cable, everyone could make a cable, and who wasn’t able to do so, had one crimped by us, for one beer (we accepted energy drinks, too). We drank a lot…
The organization had it’s flaws – some things weren’t clear before going there, we didn’t have clear policies for some things and twice I got chewed for doing some things. The design of the network was done there, and a lot of things weren’t clear, because some of the people that made the original design weren’t there, and we had a lot of misunderstandings, which we managed to clear :) We in fact didn’t have a problem that we couldn’t solve – in the beginning we even had a fiber cut – turned out that the Dutch railways went on to mow the grass with some enormous beast, and managed to remove about a meter of the cable (I can’t find the photos from the fixing, they’re worth seeing). Then, in the rain tools were found and some people went to splice the cable (we were worried that they could drown in the ditches, the rain was that heavy).
As a small conclusion – I found a lot of interesting things there, like that I’m able to sleep while it’s light, at cold, wet, in a room with 20 machines and three air conditioners, to sleep for 3 hours and again to be able to do my job, to walk in a heavy rain and not care how wet am I – in general, what doesn’t kill me, makes me stronger (and the whole event was almost murderous). Combining this with all of the information, problem, solutions, etc, I think I know why is it done only once in 4 years…
And I’ll go again in 4 years, if I’m able.
TODO:
To ask Maxim on the IDS he recommended.
To finish the mirror of all video recordings.
To do a copy of the wiki because of the rumor that it will go away in a month.
To remember some other interesting moments that I’ve missed here.