doubleTwist - iTunes for Android

[cyberface]

So I've had some issues with Apple Lossless music files on my Nexus One, and just getting the music across to the Nexus One is a bit of a pain in the arse (remove SD card, find adaptor, connect to Mac, find suitable folder, copy files across, unmount, try out in Nexus One, etc.) - no 'sync' of playlists possible as yet.

Someone pointed out doubleTwist which is meant to do this job - iTunes for Android, if you like. I'd normally be really wary of this type of software, but apparently Jon Lech Johanssen (aka DVD Jon) is behind it, and he's one of the good guys WRT media freedom and DRM, so I've downloaded and tried it out.

It relies on the USB filesystem driver for the Android devices. On the Nexus One, you can plug in via USB, then swipe down the USB notifier, press 'use USB storage' and the SD card inside the Nx1 will mount on the Mac as a writable mass storage device.

However, it doesn't. It crashes ALL the time. I can't copy more than a couple of hundred MB of data without the Nx1 crashing hard. Not an app crash, not a glitch and unmount, but a full-on kernel panic, and device hard reboot. On more than one occasion I've had the device tell me the SD card was 'potentially corrupt' which caused a momentary panic - I ripped it out, stuck it in a USB adaptor for my Mac, and had to dig through the hidden folders via the Terminal and judiciously rm -rf some evil stuff. This was *not* funny, especially when both my cats are in the vets recovering from their neutering operations and I'm anxious as HELL.

Anyway doubleTwist looks OK, though very beta and loads of obvious UI functionality is missing (expectations of a Mac user, I guess) - but with DVD Jon I expect most of the code is there to convert from one media format to another, and to do it very quickly and efficiently. This is what I want - something to take my playlist for the Nx1, sort through the media formats, convert any Apple Lossless (which the Nx1 can't play) to AAC or MP3 (which it can), etc. and then load the files onto the device. And then sync when I change the playlist in the future.

It's a lot to ask, and doubleTwist appear to be trying hard to deliver. They've got Amazon in on the act, using the app as a portal for Amazon to sell music, books and video, which is a good idea but I'd like to be able to turn it OFF as it's not my primary (actually it's not even *any*) use of the app. But I guess DVD Jon needs beer tokens. Fairy nuff on that point - nothing is forcing me to buy Amazon media through doubleTwist.

The app looks nice, and it also recognises my iPhone mk1, my iPhone 3GS, my iPad (should I say mk1 at this point? ), and my Nexus One. All Apple devices are labelled 'read-only' (I guess Apple let DVD Jon and Amazon market this app as long as it didn't directly compete with iTunes for Apple devices, again fairy nuff) but you can pull the tunes off them, by the looks of things.

BUT...... and it's an ENORMOUS BUT.... with the Nexus One crashing HARD every time I try to copy files onto the device via the USB connection, it's fundamentally unusable.

Now the crashing issue isn't anything to do with doubleTwist - I've mounted the SD card in the Nx1 manually on the desktop and tried to empty trash, and the Nx1 has crashed hard. 20 times, until I gave up, removed the card from the phone, and got all medieval with the filesystem via the Terminal and a spare USB SD card adaptor. The problem is definitely either in the Nx1 ROM (kernel, USB mass storage kext / module (I guess it's a module, being Linux...)) or in the Mac USB mass storage driver. Now I've got a lot connected to my Mac via USB and usually if a USB mass storage driver fails, or a USB device goes haywire, *everything* gets affected on OS X. But it's not. Everything else is working perfectly.

So I reckon the Nx1 ROM has a vicious bug that's causing me a LOT of grief. Could any of you HTC 'Desire' owners try this out please and see if it works? Or Nexus One owners on Windows. It could be that I need a newer ROM on my Nx1, it could be that the Mac USB driver is buggy as hell and I need to either directly write to the SD card, or build a disposable Windows box just for this (which would be perverse, given that I'm using Android and Mac to get away from Windows).

I'm losing it big style... one of my cats is showing signs of having a few issues and I'm worried as hell - and now this BS with the Nx1 is the last thing I need. I'm off to the vet, but if any of you intrepid Android users with recent (2.1) devices who have tried doubleTwist could help me out and test it - just to make sure it's not just my Mac, then I'd much appreciate it. Also, if it's a 'are you on the latest ROM?' type query - please let me know how to upgrade to the latest ROM without losing all my finely tuned user settings (getting email working properly required hours of work and I really could do *without* losing this setup)...

I have a rooted Nx1 and the bootloader with the recovery software - so tell me how to do a full backup which I can store somewhere on my Mac and restore from if necessary, and I'll get on it. Once I'm back with my cats.

[Frederick]

Just tried it myself, it's working fine although not impressed it's auto syncing everything over... I used to have a 4gb SD card, I now have a full, 4gb SD card!

Windows 7 Pro x64 and Nexus One stock & locked running 2.1-update1 / ERE27 using the OE Nexus One micro USB cable

[Frederick]

OK, after playing around with it a bit more, I really don't like that application... There's a few reasons.

I shouldn't have to create an account to use it unless I intend to purchase songs through it.

The music management side of it is horribly broken - unless every MP3 you have is tagged to perfection, it won't find it - even so far as to disregard it being there at all.

If you add music to your phone outside of the application, it steadfastly refuses to find *any* media on the device at all, even stuff it's previously synced over.

Automatic syncing with no prompting is not cool...

No id3 tagger built into the application, then if you tag songs that aren't tagged, it won't find them until it does a complete library rebuild which can take ages.

Queries the iTunes library on start for changes, and takes ages to do so.

Don't get me wrong, it has potential, but for now I'm going to stick to using Windows Explorer to put what I want onto my phone... I don't want to listen to reams of crap I've acquired over the years on one applications say so!...

[marshalla]

Not sure why you're having so much trouble - unless it's a Windows porting/Nexus issue. I use it on MacOS to sync my Pre and it only evers transfers what I want it to.

[cyberface]

What seems odd is that my Nexus One is crashing constantly under heavy USB file transfer load.

I haven't even tested yet to see whether doubleTwist is actually doing what I *hope* it's doing - ensuring that the iTunes music files in the playlist I've selected are converted to a format that Android actually understands (in my case, this is entirely Apple Lossless... all I really need is some batch script to convert all Apple Lossless files to AAC at a decent bitrate and copy those files to a certain area, and then I could just fill the Nx1 SD card manually - not a problem).

If, at the end of this, after numerous crashes, this doubleTwist app actually manages to copy the entire playlist over to the Nx1 - and then I find that more than half of the songs are still in Apple Lossless and can't be played by the Nx1 - (by the way - BOOM - the Nx1 has hard crashed again, that was a new record of 4 minutes between hard crashes) - then the app is worse than useless, it's evil as it requires an account AND doesn't do what it says it does.

I must say I'm surprised at this sort of thing from DVD Jon... but you never know, I guess...

[Frederick]

4 minutes between crashes???? it sounds up and down more times than a prostitues knickers!!

That's got to be some kind of rogue application taking the system down with it, have you pulled the error logs to see why it's kicking it's legs up?

There's an app called "Log Collector" on the market, which might give a bit more insight into what is making it crap itself, could be worth a go?

[RizzoTheRat]

My only complaint with it is how long it takes to scan my iTunes playlists. I use iTunes to play music on my PC, and transfer to the iPod, but am using doubletwist to sync a few selected playlists with my HTC Desire. Seems to work pretty well in this role, but if I've added some new music to iTunes or modified a playlist I have to rescan iTunes which takes 15-20 minutes ( 45GB of music on a 3.4GHz Phenom X4 running Win 7 64)

[cyberface]

Frederick said:

4 minutes between crashes???? it sounds up and down more times than a prostitues knickers!!

That's got to be some kind of rogue application taking the system down with it, have you pulled the error logs to see why it's kicking it's legs up?

There's an app called "Log Collector" on the market, which might give a bit more insight into what is making it crap itself, could be worth a go?

Yup I've already got an app called aLogCat.

Using your sensible suggestion (you can tell I've not been doing proper development for years now, can't you... would have been the first thing I'd done 10 years ago, now the first thing is to rant on a car forum... nutter) I waited for the device to reboot (after crashing). Waited for everything to tidy up, then started aLogCat (which shows all the logs, from which app, and colour-codes the severity-level), enabled USB debugging *and* USB mass storage mode (both of which showed up on aLogCat), then kicked off the doubleTwist 'Sync' (which I hope continued from where it left off).

The doubleTwist app was merrily converting (or doing something, anyway) as the OS X console logs showed the music file checking / ID3 tag conversion, etc. The aLogCat output on the Nx1 was ordinary - every so often an information message when then WiFi changed power state, every so often a VM garbage collection notification. Nothing abnormal looking - at all. And then the screen goes black, the Mac complains about not unmounting the volume, and the Nx1 is gone. On reboot, looking at aLogCat (infuriating app, which autoscrolls to the latest log item, even when you specify autoscroll=false and pause=true - you can't see early events for any length of time before a new log entry throws you to the end of the file. Grrrrrr) there's nothing obvious.

The only OS X error is when the device bombs - a kernel warning stating that disk7s1: media is not present - well no st Sherlock. It really doesn't look like OS X is booting the Nx1 off the bus though - it's the Nx1 bombing, and OS X having to clean up after it...

Really perplexed by this. Perhaps I'll work on the Lossless files some other way, and go play with my new iPad instead.

[cyberface]

Right - forget it completely.

It doesn't work. It is useless, and doesn't do what it claims to do.

I've given up on copying the large playlist I had built, and concentrated on just the few hundred songs I'd managed to sync with the Nexus One. Out of those, maybe 10% were originally Apple Lossless on the Mac (on my iTunes Library) - and I was expecting doubleTwist to spot this, convert the file to AAC, tag with the metadata already in the Lossless file, and copy across to the Android device.

So I check on the Music app... and nope, these files are still in there as 'Unknown Artist' - with the album art, it must be said - and when I click on one of the songs, I get the error 'sorry, the player does not support this type of audio file'. So doubleTwist doesn't convert the files at all - it merely tries to copy them, and it fails at that.

bks, I've wasted a large amount of time here. I need to write a script to loop through all the tunes in my playlist, check to see if they're Lossless, if they are then convert to AAC, if not leave, then copy the resulting file to a new location and *not change my existing iTunes Library* - I don't want to convert my Lossless files to lossy formats just so I can play the damn things on my Android device.

Interestingly enough the latest iTunes version has an option to convert all music to 128 kbit AAC if you load music onto your iPad. So I *could* theoretically hijack this process if I can make my Nexus One look like an iPad....

Any other suggestions welcomed. For now though, doubleTwist is being uninstalled and damn, they've got my email address. Punks. I'm pissed off with this, really pissed off.

[Frederick]

cyberface said:

For now though, doubleTwist is being uninstalled and damn, they've got my email address. Punks. I'm pissed off with this, really pissed off.

I feel your pain, luckily I didn't use my gmail account I use on my phone, instead I used my common or garden Hotmail account that I can add any crap to the filters and let Thunderbird take care of the rest.

You have a 16 or 32gb SD card don't you? I wonder if some of the blocks are failing, which leads to a big problem with read/write? It could be bombing the phone if you're running app2sd (which is installed as standard on some rooted/custom roms) and something happens to the 'sanity' of the card, maybe causing mount problems...

I guess if I had my logical hat on, what I'd try to do is thus:

1: Pull the SD card and refit the 4gb one, proceed to dump a couple of gigs of stuff on it and see if it fails. If it does, move on, if not - try another card, lather rinse repeat, and deduce from there.

2: If it's failed again on a different card, I'd pull the card, stick it in a USB reader and whack it into my PC - stick a few gigs of files onto it that way, then slap it in the phone and see if the phone fails to read it, or see if it segfaults again. If it reads fine, then it could be a problem with write back caching or some other black magic like that/

3: If it fails with a pre-loaded card, I'd try running the phone without a card. I don't know if the Nexus One will boot without an SD card though, as I've never tried it.

4: Last but not least, I'd do a "nandroid" backup of the device, and give it a hard reset. You really want to assess the problem using either a Linux box or a Windows PC before nuking the phone though. If fail behaviour is exhibited on the Mac only, then it's probably a driver or caching issue (some wierdness going on with regard to how the Mac is sending and verifying the data maybe?)

I guess another way to properly sanitise the SD card would be to stick it in a reader, attatch it to the PC and fill it with a text file (could take a while with plus-sized cards). Then, copy the file off it again and inspect for any changes (I'd use "fc" in a PC command prompt, god knows what in Mac). I will admit though, the more I think about it, the more I think it is a caching issue but possibly on the phone rather than the PC.

My reasoning behind this is that the phone is working as a USB card reader, in essence - but data still goes through the underlying subsystem of the phone. If (for arguments sake) a cache internal to the file system on the phone has a panic, and overflows - it could be enough to make the kernel say "wtf!??!11oneone" and panic, pulling the plug.

I'll test my N1 with some big files (couple of gigs or so) to see if any shut down issues show their faces, if not - it would point to it being a rooting/permissions/hacked around with till it barely makes phone calls issue as opposed to an inherent fault.

[Frederick]

Well, 7 minutes into a transfer of a 3.6gb file, the phone is exhibiting a little bit of a protest - instead of it illuminating the screen when you press the lock button, it just flashes the bottom row of softkey lights on for a sec, then turns them off - it's still transferring though, so we'll see if anything else happens...

[Frederick]

Ok, file transferred successfully - phone is back up and working as it should do, file nestling on the SD card and everything seems hunky dory.

I'm using the standard 4gb one that came with the phone incase it matters

[cyberface]

Frederick said:

Ok, file transferred successfully - phone is back up and working as it should do, file nestling on the SD card and everything seems hunky dory.

I'm using the standard 4gb one that came with the phone incase it matters

The first approach (before I found out that Android wouldn't play Apple Lossless by default) was to bung the SD card into a USB reader and just copy the files on. It's slow (presumably the microSD cards are one chip, one channel things purely because of space restraints) but it worked. It's only since I've tried the USB-cable mass-storage-device mode that I've seen any trouble at all.

And it's not just doubleTwist - deleting files from the mass-storage-device SD card (in the phone) and then 'emptying trash' on the Mac *still* causes Android kernel crashes. This is odd, because 'emptying trash' isn't a large file volume operation. Sounds to me like you're a Windows dev - this is the unix equivalent of the 'Recycle Bin' and all you're doing when you delete files off a Mac volume and then 'emptying trash' is to mark the files as deleted and then free the blocks. When I just tried to delete 4 GB of data from the card, there's no way that 4 GB of data was touched - all that was touched was the filesystem. And since all these Android phones appear to use bog-standard microSD cards, which come formatted as FAT32, then the only thing I'm touching in my big 'delete' is the FAT itself. Which is not big.

AFAIK the FAT filesystem 'standard' may be MS proprietary but it's well enough known these days for all operating systems to deal with it properly. Linux and OS X both read and write FAT32 perfectly happily, since most USB flashdrives are formatted FAT32 and tend to stay that way, until you need to store files bigger than 2 GB (or is it 4 GB? - well with 32 GB USB flash drives being cheap and plentiful, that limitation *is* an issue these days).

But AFAICT there's nothing on the Android microSD card that either runs into FAT32 limitations, or should be bothered by FAT32 behaviour. It's weird. If there was some simple bug in the operating system, and the FAT32 filesystem driver didn't play well with multiple kernel processes (filesystem drivers should be bulletproof. There's no excuse for this in production systems), then I could understand the big crashes. But it's not as if some module gets all screwed up - I've seen this happen on OS X plenty of times, and usually one process fails to exit, knocking on to other processes, and eventually the whole system grinds to a halt unless you can shell in and kill -9 1 fast enough. But a full, immediate system crash (aka kernel panic in OS X and most Unix) - that's a big deal, and I haven't seen one on OS X for many years. And the ones I've seen that *weren't* caused by bad hardware or bad third-party kernel extensions - IIRC only early 10.0, 10.1 and some situations involving network or Firewire drives 'disappearing' in 10.2. It's the Unix equivalent of the BSOD.

I'm confident that simply sticking the microSD card into my tiny USB interface and copying the files across using the standard Mac system will work just fine. I was hoping that doubleTwist would do the hard work of transcoding the troublesome Apple Lossless files for me, and not result in an iTunes Library that's stuffed with multiple formats that I don't need *in* the Library. Since it doesn't, and the latest version of iTunes seems to have an automatic 'transcode to 128 kbps AAC for iPod / iPhone / iPad' facility, I may try to hack that. Sadly iTunes won't sync to my Android phone but I can get it to sync to a spare 32 GB iPhone 3GS, and then FTP all the music files off the iPhone and onto my Nexus One. The obfuscation that the iPhone uses won't bother Android, which just scans the music folder and uses the metadata *like it should* and hence can use iTunes Library Folders with *no modification*...

I'll try that next

[Frederick]

Emptying trash / clearing the recycle bin etc really honestly shouldn't cause any major hassles to anything as it's just flagging that space in the file allocation table as being writeable - so yeah, negligable volume impact...

Have you tried sticking another card into the Nexus just incase there's a problem with the big one you have in there at the minute?

I'm totally at a loss to understand why it's rebooting though - using the Nexus as 'USB Storage' isn't doing much more than using it as a glorified card reader.

[sadako]

The crashing issue under usb mass storage might be a nexus one only issue. I have never had issues doing large file transfers from windows and linux on my G1, or smaller ones from osx.

[lawrence567]

I have a lot of music in my itunes.
A lot of it ripped straight to Itunes from CD or purchased through Itunes.
Is there anyway to drag & drop directly from Itunes of can i only drag & drop if it's saved elsewhere on the comp?

[cyberface]

Thought I'd update this thread to keep it in one place.

It appears that the USB mass storage thing is a problem either with the Nexus One, or with my Mac. Whilst I'm always quick to defend OS X, there's always the possibility of duff hardware in the Mac Pro that I'm doing this with. However, I've just synchronised 14 GB of music with both a spare iPhone 3GS and my new iPad, both over USB, so the chances of the USB port on the Mac that the hub these devices were connected to being faulty is pretty slim.

Looks like a Nexus One-only issue, or at least an Android SDK 2.1 / Mac OS X issue. I haven't tried using the Nx1 as a USB mass storage device on a Mac without installing the Android SDK... remember with the SDK on the Mac, there is the adb process running. But I specifically killed all SDK daemons when testing, so the Mac side is likely to be OK.

And, remember, I've got a hacked rooted ROM on my Nexus One - it may be out of date, it's one of the MoDaCo ROMs. I need to do something about this.


Anyway, regardless, the bottom line is that doubleTwist doesn't actually do what it claims it does. What I need, from an 'iTunes for Android' perspective, is something that will accept a playlist or folder of media files (in formats that are playable by the host operating system, in my case OS X), convert them to formats that are playable by Android, ensure the correct metadata is included in the converted file, and then copy them onto the Android SD card.

It's clear already that for the majority of users, just copying the iTunes Library 'Music' folder from your Mac (or Windows box) onto a root-level folder on your Android SD card will work perfectly, because iTunes bungs the metadata (artist, album, track number, album art) in each file for you, and Android scans the folder and builds what looks like an SQLITE database (much like the iPhone) with the metadata inside each media file. So if your iTunes folder is 'organised' by iTunes, then each file will contain the essential items of artist, album, track number, track name - and Android will be able to show these files as being part of an album from a particular artist. iTunes also puts the album art in each file - Android also picks this up. It's really rather cool.

The problem I had was simply that I have a bunch of files in my iTunes library that are Apple Lossless format. Android doesn't support this format, and the presence of such files (they have an .m4a extension just like AAC files) causes confusion in the Android music player. I needed some way of converting the Lossless files into high-quality AAC so my Nexus One would play them.

The idea was that doubleTwist would do it - I encountered issues with the USB sync - but regardless of the USB problems, doubleTwist does NOT convert Apple Lossless m4a files into AAC m4a files, meaning it adds NO value over me just copying the iTunes Library across onto the SD card. The Lossless files still can't be played.

So I've had to come up with another solution. I don't want to convert all my iTunes Library to AAC because I'd end up with duplicate songs and a LOT of stuff to tidy up - it'd take ages. Maybe some day I'll work out a way to rip *everything* to an open lossless format (WAV? AIFF?) and then store a copy of each file in original lossless, AAC, and MP3 - and then use 'smart playlists' to filter the files so I can play the right type of file on the right sort of equipment. But I haven't got there yet.

What I *have* done, and this appears to work, is to go into iTunes, ensure the 'keep iTunes folder organised' flag is set, change the Library folder to a temporary empty folder (for copying to the Android SD card), then select all the tunes in the playlist I want to transfer to Android, and use the menu option 'Create AAC version'. This would normally duplicate your files in the Library... but because I've changed the location of the Library, the 'new' created AAC files are put into the new temporary 'for Android' folder. They're also organised by artist and album, but more importantly all the metadata is preserved in each file (including album art). Once this is done, I've got a full folder of files for my Nexus One - just the playlist I asked for, all converted to playable AAC, organised in the filesystem but also containing all the metadata.

It's just completed the conversion process, for 15 GB of data. So I'll have to cut some out, and then copy to my Android SD card. Then, I'll have to reset the iTunes Library location back to the original, and clear up any duplicate entries in my Library. But the important thing is that I now have a folder full of AAC versions, complete with metadata, that will work on my Nexus One.


Not an ideal option as it involves a bit of work. But certainly one for anyone else here with my problem....

[lawrence567]

lawrence567 said:

I have a lot of music in my itunes.
A lot of it ripped straight to Itunes from CD or purchased through Itunes.
Is there anyway to drag & drop directly from Itunes of can i only drag & drop if it's saved elsewhere on the comp?

bump about my above post...
any news?