peterlog

the web log of peter sabaini

  • Home
  • About
    • 0
      22 Jan 2012

      Puppet & Ruby

      • Edit
      • Delete
      • Tags
      • Autopost

      So I spent the last week in Amsterdam for training. Amsterdam is a great city, so it's unfortunate that I didn't have all that much time for exploration. Luckily I at least managed a small trip to the Rijksmuseum; while most of the museum is closed for restoration, they have a small selection with truly great pieces on display.

      Ijfafjjd

      Training was also very enjoyable however. Topic of the training was the IT automation toolkit Puppet, and, since Puppet is written in Ruby, also a little of that was covered. Mr Haugen from Puppetlabs did an excellent job as an instructor; with just the right mix of theory presentation and exercises (and in between answering lots of questions).

      I have a bit of a love/hate relationship with Puppet. I've been using Puppet before; it sure is a very useful tool but on the other hand... It does have some warts.  

      For one the terminology of the DSL. For someone who has done OO programming the word "class" carries very specifc meaning. Puppet also has classes, but the are almost entirely unlike those from OOP. For example, all Puppet classes are singletons and may not be instantiated more than once. Also, there is some inheritance but overriding parameters is not supported; so the advice is to avoid inheritance and use includes. Include is not at all like someone with a C (or Erlang) background might expect it to be; it's not a simple literal file include but actually a kind of import + instantiate function. And so on.

      Jbiebdfj

      On the other hand, I much like the declarative model and the fact that it is very open and extensible. For instance, configuration parameters need not be held in static files but may also reside in an external repository; all that is needed is a glue script to fetch said parameters.

      Also Puppets approach is refreshingly pragmatic; it's all about getting stuff done. One can easily see that there are actual sysadmins using this stuff in production. It has decent support for testing and dry runs, for example -- good to have if you're not that confident about a system change you're going to roll out to a few thousand hosts. 

      It actually reminded me a little bit of the earlier days of Zope2 -- lots of magic and little documenation; for someone who knows how to push the right buttons a lot can be done with little effort but to get there you have a not so friendly learning curve.

      Puppet is written in Ruby, and for many types of extensions you have to get down to the Ruby level.

      Bcejeddf

      I haven't done much with Ruby before, so I'd like to take up the opportuniy write up some thoughts about the language and platform. My reference here is mainly Python, a) because these languages are in a pretty similar niche and b) Python is what I know best.

      For this Python programmer, Ruby tends to feel a little "Perlish" from time to time. In cases where Python might choose to focus on "one right way" to do something, Ruby would rather provide several ways (in different flavors) to accomplish the same thing. This sometimes can be quite useful, sometimes rather confusing. For me, this makes Python more intuitively useful, but admittedly sometimes more verbose. 

      One of the Perlish features of Ruby that I quite like are built-in regular expressions. For Sysadmins who constantly have to wrangle output of other tools to their own end regular expressions are a must have.  While Python of course has an excellent library for RE's it is still not the same level of integration as a native datatype with accompanying operators etc. On the other hand Ruby has Perls habit of setting tons of special global variables for RE matches -- not trying to be dogmatic here but can we maybe just try to avoid automatic global vars? Please? People get hurt by that stuff! This, to be fair, is entirely possible because Ruby _also_ provides other ways of accessing these values, without resorting to globals.

      One of the Perlish things in Ruby that I wish they just had left out is the postfix conditional, ie. one can write "if x then foobar()" but also "foobar() if x". Larry Wall has stressed how that helps make Perl like writing English but in all honesty I'd like to have my software a little more regular than the English language with its abundance of corner cases and special rules for all kinds of occasions.

      Speaking of conditionals -- many things which would be statements in other languages are actually expressions in Ruby, ie. they have a value; also including if-conditions. Eg. this function would add 2 to numbers smaller than 23 (since functions return the value of their last expression):

             def foo(a)
                 if a < 23
                    a+2
                 else
                   a
                 end
             end

      I find having more expressions lets one focus more on the data than on the procedure of how to operate on it. Yay for expressions! 

      Ruby has a very easy way to parametrize methods with anonymous blocks of code. While this allows for some very higher-order / functional coding style (which I appreciate) I had the impression that feature tends to get somewhat overused (well at least from what I've seen in Puppet). Sometimes it felt as if people we're actively trying to avoid writing regular methods and instead resorting to fiddling with blobs of anonymous code to achieve the same thing. Still, good to have for functional programming.

      Ruby is more consistent in its OO compared to Python, eg. while Python provides a built-in "length()" function, in Ruby only values which actually have a length (ie. Arrays) provide a ".length()" method. Makes a lot more sense, in my opinion.

      By the way, you may call methods with or without parentheses in Ruby. Personally I prefer parentheses for calls, I find code easier to read that way, but that might also just be a matter of practice.

      As to concurrency, Ruby unfortunately has similar problems as Python. The only directly supported concurrency model is a threading one, and like Python Ruby has to serialize access to data through a global interpreter lock. This usually results in Ruby code not scaling up very well on CPU-bound tasks. On the other hand, if you are doing CPU-intensive tasks you might want to consider a C extension anyway; these are luckily quite easy to embed (there even is a library that lets you inline C code into Ruby).

      Iijgcbbj

      Moving away from the core language on to the runtime environment -- there's a lot of good and also some bad there. 

      First to the bad: the standard library is, at least compared to the Python equivalent, a pretty rough place. Many modules are sparsely documented or not at all, and for some of the modules I kinda
      wonder if anyone uses them.

      The good, of course, is the gem system, to the point that I'm thinking maybe gems are the reason the stdlib seems to be lacking in maintenance.

      Gems is Ruby's packaging format and application, and it handles everything a packaging system should -- installation, dependency management, querying, etc. Pythons packaging system(s) are, in comparison, a bit of a mess.

      There are a lot of gems available. Rubygems.org boasts a total of 33160 packages available, and >444 million downloads. While nothing can be infered about the quality and/or usefulness of the provided libraries (and while still below CPANs legendary scale), these are nevertheless some impressive numbers.

      All in all, I'm looking forward to doing more Puppet and Ruby. There's lots to discover!

      • views
      • Tweet
    • 0
      6 Feb 2011

      More dynamic Python imports

      • Edit
      • Delete
      • Tags
      • Autopost

      I like the idea of this — make Python imports more dynamic through an explicit mapper, as implemented by Exocet

      I’ve implemented plugin systems previously myself, and it’s not that hard in Python, but with Exocet its much cleaner (and with more features) than what I did.

      • views
      • Tweet
    • 0
      23 Dec 2010

      21 Lines Spelling Corrector

      • Edit
      • Delete
      • Tags
      • Autopost

      Mr Norvig demonstrates a neat spelling corrector in 21 lines of clean Python. Good explanation of the statistics too, and an updated section with implementations in other languages.

      • views
      • Tweet
    • 0
      13 Aug 2010

      emkuu on github

      • Edit
      • Delete
      • Tags
      • Autopost

      My thesis project, a RESTful Messagequeueing middleware called emkuu, now lives on github

      • views
      • Tweet
    • 0
      30 Jul 2010

      My Python History

      • Edit
      • Delete
      • Tags
      • Autopost

      As a Siemens intern in summer ‘96, my task was to process megabytes and megabytes of log files from network testing equipment, using the not-so-formidable text parsing capabilities of Visual C++ 1.5. After this rather grueling experience, I was on the lookout for an easier way to do string processing and stumbled on scripting languages. This was my first encounter with Python. I’m sad to say I was among those turned off by whitespace-as-indentation. Nevertheless, I was electrified by scripting, and Perl became my tool of choice for text processing (and soon for everything).

      Two years later, increasingly frustrated with Perls inscrutable syntax and somewhat cumbersome OOP syntax, I started to tinker with Python again. This got a kickstart when I was introduced to Zope. Zope impressed me from the start — finally a structured approach to the chaotic world of CGI and HTML generation. I proposed to port the online edition of the newspaper I worked for to Zope, and much to my own surprise, got a green light.

      I’ve been working with Python since then, often with Zope, but also with Twisted, a bit of Plone and the ZCA.

      Although other languages have kept me busy lately, I still consider Python my main language.

      • views
      • Tweet
    • 2
      12 Jul 2010

      Filesystem Brutality

      • Edit
      • Delete
      • Tags
      • Autopost

      Or, how to scrape data off a broken harddrive with unnecessary cruelty

      The other day a friend of mine sent a terabyte harddrive by which, due to events unknown, had ceased to perform its duty in a Linux-based NAS.

      Indeed, it seemed as if the disk in question 1) indeed contained an ext3 filesystem, and 2) had lost its partition table, and as no information about the partitioning was available, I started off to try the usual tools to guess a layout. I first tried GPart , then Testdisk, both without much success.

      I proceeded to try and find a partition table manually; as there are usually backup copies of this table present. Manual search, however, turned out to be quite tedious, so I searched for a way to automate this.

      I came up with this little Python thing:

      from subprocess import *
      
      def losetup(offs):
          cmd = ['/sbin/losetup', \
                 '-o', str(offs), \
                 '/dev/loop0', '/dev/sdb']
          call(cmd)
      
      def fsck(n):
          print "\nTrying byte %s" % n
          cmd = ['/sbin/fsck.ext2', '-n', '/dev/loop0']
          p = Popen(cmd, stdout=PIPE, stderr=PIPE)
          out, err = p.communicate()
          if p.returncode != 8:
              print out
              print err
      
      def delo():
          cmd = ['/sbin/losetup', '-d', '/dev/loop0']
          call(cmd)
      
      for n in xrange(0, 4096):
          losetup(n)
          fsck(n)
          delo()

      It really is a bit brutal. It iterates through the first 4k bytes of the harddrive (I made a backup image beforehand) and tries to perform a fsck (without actually changing anything). I basically let fsck do the test if a partition table can be found (or a backup). And, much to my surprise, fsck actually found a backup partition table using an offset of 512, so all I had to do was losetup once more, and this time perform the fsck for real. Almost all of the data could be recovered! Yay losetup and Python!

      • views
      • Tweet
    • Search

    • Tags

      • programming
      • linux
      • system
      • switzerland
      • photo
      • posterous
      • zug
      • database
      • android
      • erlang
      • kitteh
      • music
      • tool
      • couchdb
      • electronica
      • teaching
      • work
      • JavaScript
      • NoScript
      • animation
      • aquarium
      • austria
      • biking
      • bug
      • comic
      • conveniencetrumpssecurity
      • dns
      • ehealth
      • elga
      • emacs
      • emkuu
      • fun
      • gaming
      • gesundheitspolitik
      • golang
      • graphic novel
      • html5
      • industrial
      • java
      • latex
      • mail
      • messaging
      • microformats
      • mysql
      • oracle
      • perl
      • politics
      • postgresql
      • puppet
      • rant
      • restructuredtext
      • ruby
      • science
      • security
      • skype
      • styria
      • twisted
      • vintage
      • younggods
      • zeroconf
      • zope
    • Archive

      • 2012 (13)
        • May (2)
        • April (1)
        • March (2)
        • February (1)
        • January (7)
      • 2011 (29)
        • December (5)
        • September (4)
        • August (3)
        • July (1)
        • June (1)
        • April (4)
        • March (5)
        • February (1)
        • January (5)
      • 2010 (35)
        • December (6)
        • November (4)
        • October (3)
        • September (5)
        • August (5)
        • July (11)
        • June (1)
    • Obox Design
  • peterlog

    software, sysadmin, python, zope, erlang, linux, teh interwebs, cats, austria, switzerland

    18898 Views
  • Get Updates

    Follow this Space »
    You're following this Space (Edit)
    You're a contributor here (Edit)
    This is your Space (Edit)
    Follow by email »
    Get the latest updates in your email box automatically.
    Loading...
    Subscribe via RSS
    TwitterFacebookBuzzLinkedIn