Latest Diaspora* update was a lot of work

So I’ve had some issues with Diaspora*, but the main one is inter-pod communication which doesn’t seem to work back to the main joindiaspora.com pod (and some others). I had neglected updating for a little while and thought I should get to it, and see if this issue was resolved.

**Caution – the rest of this discussion is fairly technical **

I had some trouble with the oath2 ruby gem for some time now as well, and during the update process I kept receiving errors about the .gemspec file format. I went through the usual update process to see if there would be an update to oath2 amongst others. So I used my usual 3 update commands:

git pull origin master
DB="mysql" bundle install --without development,test
RAILS_ENV="production" DB="mysql" bundle exec rake db:migrate

to find that it was more upset than usual about the update. The last command didn’t want to run at all, and there were far too many errors to see what was going on.

The easiest solution to this I thought would be to clean up the Ruby gems by updating everything and seeing what happened. To do a more thorough udpate of the gems, I ran

bundle update

which resulted in:

WARNING:  Invalid .gemspec format in '/usr/lib/ruby/gems/1.8/specifications/oaut
h2-provider-0.0.19.gemspec'
Gems included by the bundle:
Could not find oauth2-provider-0.0.19 in any of the sources

So I tried

gem update

and that’s when it all got interesting. Apparently I had run out of disk quote. Disk quota? As root? I check quota, sure enough, no quota applied. I ran a df to find I was using only 23% of disk space. So I tracked it down in the end – out of inodes.

The magic command

df -i

indicated that I’d used 100% of inodes. So then my search to find out where I could get more inodes from began.

It turns out you can clean up some old files from git by issuing a command to do a garbage collection:

git gc

of course, you can’t do this without having *some* spare inodes. So I cleaned up some otherwise unused files from a few places. Running this command gave me 25 spare inodes. Hmmm… still looking…. Then it turned out that you need to ask gem to do some housekeeping. Gem keeps all the previous versions of RubyGems, and isn’t shy of using a lot of files. The magic words for gem are:

gem cleanup

which gave me over 20,000 free inodes after much work, ignoring a lot of errors and saying no to any extra removals that it questioned. Much better! Then I could complete the gem update I tried earlier. Sadly, there were still a lot of errors and it made it difficult to see what was going on, so I went about how to fix the invalid .gemspec format error. Fixing this required an update to gem itself, which can be run with

gem update --system

another command which may be useful is

gem install rubygems-update

running this solved the .gemspec format error, but now 4 of the gems complained about an invalid datestamp format inside the .gemspec. This also needed fixing, which can be done by executing

gem pristine

and then updating back to current versions, followed again by cleaning up the unwanted files. You can do all of them at once, e.g. ‘gem pristine oath2 faraday mini_magick uuid’ although I should mention that one didn’t work the first time for me so I had to do that step again. Then you can run ‘gem update’ followed by ‘gem cleanup’ and hey presto – totally clean install of gems!

Now it was back in business with running the system properly. I ran the db:migrate successfully this time, and when I ran the server it couldn’t connect to redis. Apparently I had killed the process and shouldn’t have. Easily fixed with the old:

nohup redis-server &

So now diaspora* is running on my server again! Sadly, I still can’t communicate with other pods, but at least I got back in to the swing of system admin again.

Diaspora* update

After some server tweaking to make better use of resources, and some code updates, my Diaspora* pod is much more stable again. Although it is written in Ruby, the footprint isn’t too large – and I am yet to resort to low memory versions of Ruby.

I have now posted from Diaspora* to both Twitter and Facebook. The interface looks more polished with every update, and new little features and bugfixes are coming along nicely. It is still very much in the Alpha stage of development, but is improving every day. The link to Facebook seems to be much more stable now. This may have been a temporary developer Facebook issue, as it would not previously allow me to enter the correct details for security.

I would also like to take a moment to remember Ilya Zhitomirskiy, co-founder and a developer of Diaspora*, who left us far too soon.

Server updates

I now have Drupal installed to serve web pages for my technology consulting profile. I’m yet to put any real content on there, but the exercise in building the website was worthwhile. I was surprised at the speed of configuration (mine is not the most simple system now), and also impressed at the availability of themes. I was also impressed at how easy it is to theme plain HTML pages outside of the CMS with freely available CSS themes.

I’ve also gone through the update process for Diaspora* quite a few times, and the instructions work well. The only note is that they should also point out it might be a good idea to update gems at the same times.

My Diaspora* Pod is up!

As a further foray in to open source software, and taking a more personal ownership to sharing data, I’ve had a look in to Diaspora*. Rather than dabbling round the edges, I’ve installed my own pod.

If you are unfamiliar with Diaspora*, what it is and what it stands for, check out Join Diaspora*.

So installing it is far from easy. It is still considered to be in “Alpha”, and so instructions are changing on a regular basis. Also, as it can integrate with other products such as Facebook and Twitter, instructions on how to do that has to keep pace with those products and their APIs.

Simplest and best advice for installation is follow the instructions on the GitHub wiki for Diaspora*, and it will get you a long way. There are plenty of sites telling you the instructions are wrong and provide alternatives, but this appears to be no longer required. The next thing is to take a good look at everything in the config directory, even if you don’t think you need to configure it. I had a silent failure (Diaspora* has a habit of just stopping when things go wrong and not always providing log file comments) due to a non-existent directory being required as a log file repository.

The biggest gap in instructions for me was how to configure Apache2 to do all the things you need it to. It largely came down to working out which modules need to be loaded to fix which not-so-helpful error messages.

I’ve managed to get Twitter to allow me to twet from Diaspora*, but I’m yet to have any luck with Facebook. It seems to revolve around a lack of callback URL, something which the Facebook interface is yet to let me do. This is one of the issues where if I let my side try to contact Facebook and it doesn’t work, Diaspora* aborts ungracefully, leaving me to have to clean up the still running processes. Failure to clean up the ruby processes causes great confusion on the next restart. I’m currently running the server on a detached screen which is a little bit manual, but it seems to work.

Stats and security

While developing this blog, I’ve been working on the background on security and other curiosities. I wanted to use a log processor to look at where some of the hits on this site were coming from, so of course I turned to AWstats. I also wanted to make this viewable by myself, and didn’t want another password to either type in or store. There were great instructions available for basic installation, but I had to look a little harder for secuirity. I only really want to view the stats from home, and luckily I have a fixed IP address. This is certainly a lazy way to do security, and I haven’t yet pushed it to be SSL-only, but to bolt down the AWstats pages by IP address all that was required in the Apache2 conf file was:

[sourcecode language=”xml”]
Alias /awstatsclasses "/usr/share/awstats/lib/"
Alias /awstats-icon/ "/usr/share/awstats/icon/"
Alias /awstatscss "/usr/share/doc/awstats/examples/css"
ScriptAlias /awstats/ /usr/lib/cgi-bin/
<Location /awstats>
Options ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order Deny,Allow
Deny from all
Allow from 1.2.3.4
Allow from 127
</Location>
[/sourcecode]

Where of course the 1.2.3.4 is replaced with your desired access address – IP address or DNS address

In other security related news, somebody else ran a script against this site but at least it was a better attempt. They looked for 151 vulnerabilities in a single sitting rather than the 6 the last guys did. There’s been some other feeble one or two since then, but it’s nice to be tested thoroughly.

Newsflash: I’ve now had my first spam comment submitted. Hooray!

Raising the rank

Part of the point of this blog itself of course is to improve the quality of information that exists online about me, and outweigh the misinformation brought about by the identity theft incident. As a part of ensuring that this blog remains well ranked, I am trying to submit it to all major engines that accept submission – Google, Bing, Yahoo, Ask; some of which required me to put an authentication file in the root of the web server. A secondary part is including a sitemap.xml file for pre-indexing information. I have installed a plugin to WordPress to see if that will do the job automatically rather than me having to generate one separately and then upload it. Conveniently, the plugin advertises itself in the sitemap.xml file it generates.

Back to technology adventures – the hacking attempts begin

This was a little more overt and quite frankly faster than I expected. They were nice enough to sign it though.

[Sun Sep 18 08:09:48 2011] [error] [client 210.71.198.186]
 File does not exist: /var/www/w00tw00t.at.blackhats.romanian.anti-sec:)
[Sun Sep 18 08:09:49 2011] [error] [client 210.71.198.186]
 File does not exist: /var/www/phpMyAdmin
[Sun Sep 18 08:09:50 2011] [error] [client 210.71.198.186]
 File does not exist: /usr/share/phpmyadmin/scripts
[Sun Sep 18 08:09:50 2011] [error] [client 210.71.198.186]
 File does not exist: /var/www/pma
[Sun Sep 18 08:09:51 2011] [error] [client 210.71.198.186]
 File does not exist: /var/www/myadmin
[Sun Sep 18 08:09:52 2011] [error] [client 210.71.198.186]
 File does not exist: /var/www/MyAdmin

Well, the script was written by Romanians (according to a quick search). The list of attempted hacks seems remarkably short for a brute force scripted attack. Just for the record, the whois owner of that address is currently:
Taiwan Taipei Chtd Chunghwa Telecom Co. Ltd
although, as I know far too well now for all the wrong reasons, the whois data isn’t always useful, helpful or correct.

Already being indexed?

So apparently it doesn’t take long for someone (Google?) to find you. I don’t believe I’m linked from anywhere yet, but I did ask an online site to create a sitemap.xml for me.  With timestamps only minutes before the timestamp on my sitemap.xml file this is in my error.log:

[Fri Sep 16 13:37:39 2011] [error] [client 66.249.67.135] File does not exist: /var/www/robots.txt
[Fri Sep 16 13:49:31 2011] [error] [client 66.147.244.79] File does not exist: /var/www/robots.txt

A quick googling of those IPs seems to show that the first one is owned by googlebot.com, and the second one is an anonymous looking box on bluehost.com. Friend or foe?

Hello world!

This is indeed my first post! It has been fun getting this up and running on a Lucid build of Ubuntu.

These are some of the things I discovered so far:

  • ufw out-of-the-box prevents apt-get from working
  • installing the wordpress package via apt-get resolves some dependencies: it downloads the dependencies for mysql-client but does not include any mysql-server
  • wordpress on Ubuntu works very easily once you find the special configure for Debian files inside /usr/share/docs/wordpress
  • of particular use is /usr/share/docs/worpress/examples/setup-mysql which doesn’t quite work as described, but boy did it make everything work
  • I still haven’t figured out the timezone for this yet
  • Bitvise Tunnelier is awesome for ssh sessions under Windows7 64-bit

So far things have worked rather well! On to the next bits….