So I’ve had some issues with Diaspora*, but the main one is inter-pod communication which doesn’t seem to work back to the main joindiaspora.com pod (and some others). I had neglected updating for a little while and thought I should get to it, and see if this issue was resolved.
**Caution – the rest of this discussion is fairly technical **
I had some trouble with the oath2 ruby gem for some time now as well, and during the update process I kept receiving errors about the .gemspec file format. I went through the usual update process to see if there would be an update to oath2 amongst others. So I used my usual 3 update commands:
git pull origin master
DB="mysql" bundle install --without development,test
RAILS_ENV="production" DB="mysql" bundle exec rake db:migrate
to find that it was more upset than usual about the update. The last command didn’t want to run at all, and there were far too many errors to see what was going on.
The easiest solution to this I thought would be to clean up the Ruby gems by updating everything and seeing what happened. To do a more thorough udpate of the gems, I ran
bundle update
which resulted in:
WARNING: Invalid .gemspec format in '/usr/lib/ruby/gems/1.8/specifications/oaut h2-provider-0.0.19.gemspec' Gems included by the bundle: Could not find oauth2-provider-0.0.19 in any of the sources
So I tried
gem update
and that’s when it all got interesting. Apparently I had run out of disk quote. Disk quota? As root? I check quota, sure enough, no quota applied. I ran a df to find I was using only 23% of disk space. So I tracked it down in the end – out of inodes.
The magic command
df -i
indicated that I’d used 100% of inodes. So then my search to find out where I could get more inodes from began.
It turns out you can clean up some old files from git by issuing a command to do a garbage collection:
git gc
of course, you can’t do this without having *some* spare inodes. So I cleaned up some otherwise unused files from a few places. Running this command gave me 25 spare inodes. Hmmm… still looking…. Then it turned out that you need to ask gem to do some housekeeping. Gem keeps all the previous versions of RubyGems, and isn’t shy of using a lot of files. The magic words for gem are:
gem cleanup
which gave me over 20,000 free inodes after much work, ignoring a lot of errors and saying no to any extra removals that it questioned. Much better! Then I could complete the gem update I tried earlier. Sadly, there were still a lot of errors and it made it difficult to see what was going on, so I went about how to fix the invalid .gemspec format error. Fixing this required an update to gem itself, which can be run with
gem update --system
another command which may be useful is
gem install rubygems-update
running this solved the .gemspec format error, but now 4 of the gems complained about an invalid datestamp format inside the .gemspec. This also needed fixing, which can be done by executing
gem pristine
and then updating back to current versions, followed again by cleaning up the unwanted files. You can do all of them at once, e.g. ‘gem pristine oath2 faraday mini_magick uuid’ although I should mention that one didn’t work the first time for me so I had to do that step again. Then you can run ‘gem update’ followed by ‘gem cleanup’ and hey presto – totally clean install of gems!
Now it was back in business with running the system properly. I ran the db:migrate successfully this time, and when I ran the server it couldn’t connect to redis. Apparently I had killed the process and shouldn’t have. Easily fixed with the old:
nohup redis-server &
So now diaspora* is running on my server again! Sadly, I still can’t communicate with other pods, but at least I got back in to the swing of system admin again.