Dell XPS 8300: Kernel configuration tips

I recently installed Gentoo on a Dell XPS 8300. The following are some notes that I took regarding configuring the kernel. I’ve been using Linux for quite a while; however, I generally have not had to configure the kernel too much. Luckily, it has generally “just worked” heretofore. Unluckily, that means I have no experience/opportunity to configure a kernel.

For better or worse, we learn from our mistakes.

Using lspci to find hardware modules

lspci is your friend. Or at least it is if you run it in a non-chrooted environment. Using the –v option (or also –k) will tell you what module needs to be loaded to work with the hardware. Armed with this and LiveDVD, you should not have a problem locating what kernel modules you need to enable.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 clubby ~ # lspci -k
 00:00.0 Host bridge: Intel Corporation Device 0100 (rev 09)
         Subsystem: Dell Device 04aa
 00:01.0 PCI bridge: Intel Corporation Device 0101 (rev 09)
         Kernel driver in use: pcieport
 00:16.0 Communication controller: Intel Corporation Cougar Point HECI Controller #1 (rev 04)
         Subsystem: Dell Device 04aa
 00:1a.0 USB Controller: Intel Corporation Cougar Point USB Enhanced Host Controller #2 (rev 05)
         Subsystem: Dell Device 04aa
         Kernel driver in use: ehci_hcd
 00:1c.0 PCI bridge: Intel Corporation Cougar Point PCI Express Root Port 1 (rev b5)
         Kernel driver in use: pcieport
 00:1c.1 PCI bridge: Intel Corporation Cougar Point PCI Express Root Port 2 (rev b5)
         Kernel driver in use: pcieport
 00:1c.3 PCI bridge: Intel Corporation Cougar Point PCI Express Root Port 4 (rev b5)
         Kernel driver in use: pcieport
 00:1c.4 PCI bridge: Intel Corporation Cougar Point PCI Express Root Port 5 (rev b5)
         Kernel driver in use: pcieport
 00:1d.0 USB Controller: Intel Corporation Cougar Point USB Enhanced Host Controller #1 (rev 05)
         Subsystem: Dell Device 04aa
         Kernel driver in use: ehci_hcd
 00:1f.0 ISA bridge: Intel Corporation Device 1c4a (rev 05)
         Subsystem: Dell Device 04aa
 00:1f.2 RAID bus controller: Intel Corporation 82801 SATA RAID Controller (rev 05)
         Subsystem: Dell Device 04aa
         Kernel driver in use: ahci
 00:1f.3 SMBus: Intel Corporation Cougar Point SMBus Controller (rev 05)
         Subsystem: Dell Device 04aa
         Kernel driver in use: i801_smbus
         Kernel modules: i2c-i801
 01:00.0 VGA compatible controller: ATI Technologies Inc Juniper [Radeon HD 5700 Series]
         Subsystem: ATI Technologies Inc Device 3000
         Kernel driver in use: radeon
         Kernel modules: fglrx
 01:00.1 Audio device: ATI Technologies Inc Juniper HDMI Audio [Radeon HD 5700 Series]
         Subsystem: ATI Technologies Inc Juniper HDMI Audio [Radeon HD 5700 Series]
         Kernel driver in use: snd_hda_intel
 02:00.0 Network controller: Broadcom Corporation Device 4727 (rev 01)
         Subsystem: Dell Device 0010
 03:00.0 Audio device: Creative Labs X-Fi Titanium series [EMU20k2] (rev 03)
         Subsystem: Creative Labs Device 0044
         Kernel driver in use: snd_ctxfi
 04:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57788 Gigabit Ethernet PCIe (rev 01)
         Subsystem: Dell Device 04aa
         Kernel driver in use: tg3
 05:00.0 USB Controller: NEC Corporation Device 0194 (rev 03)
         Subsystem: Dell Device 0194
         Kernel driver in use: xhci_hcd
 

Searching for kernel options

Having been using Linux for ten years now, it is somewhat embarrassing to admit that I just learned about the search feature of menuconfig. Change directory to /usr/src/linux/ and type make menuconfig. By hitting the forward slash key (i.e. /), a search dialog will appear for simple text queries.

Miscellaneous kernel options

Some kernel options needed for makes that need to be installed:

  • CONFIG_AUDITSYSCALL: consolekit
  • CONFIG_CPUSETS: torque
  • CONFIG_USB_SUSPEND: udisks
  • CONFIG_PID_NS and CONFIG_NET_NS: chromium

Optimize for Sandy Bridge

It so happens that I am fortunate enough to be using a Sandy Bridge. If you are interested in squeezing out some more speed, try optimizing the march flag used for compiling the kernel by editing the arch/x86/Makefile file and set the appropriate march option to corei7-avx.

My Kernel Config

Here is my kernel config for a Dell XPS 8300 against the 3.2.1-gentoo-r2 kernel.

config.gz

Packages that point at the kernel

After rebuilding the kernel, you might want to rebuild these packages too.

  • libv4l
  • consolekit (CONFIG_AUDITSYSCALL)
  • torque (CONFIG_CPUSETS)
  • udev
  • udisks (CONFIG_USB_SUSPEND)
  • upower
  • lvm2
  • lm_sensors
  • chromium

Gentoo Init Scripts: Use of the opts variable is deprecated and will be removed in the future

After updating my Gentoo Linux box, I noticed this warning when some of my init scripts ran:

  • Use of the opts variable is deprecated and will be removed in the future.

The short answer is that you can probably ignore this warning for now. The long answer is that there is a new format for the init scripts and not all of them have been updated to the new format. Obviously, as a sanity check you should make sure that after the update you merged the changes (if any) to configuration files by running etc-update. This issue should resolve itself eventually by the package maintainers.

nspluginwrapper: double free or corruption (out)

If you are having problems unmerging nspluginwrapper on Gentoo Linux, here is a workaround I found in the Italian Gentoo forums.

   
  * Removing wrapper plugins... 
 *** glibc detected *** nspluginwrapper: double free or corruption (out): 0x0000000000626ba0 ***
 

Executing the following shell script with super-user privileges should remove the troublesome package.

 1
 2
 3
 4
 5
 6
 7
 8
 #! /bin/sh 

 rm /usr/bin/nspluginplayer                                                    
 rm /usr/bin/nspluginwrapper                                                  
 rm /usr/lib64/nsbrowser/plugins/npwrapper.so 
 rm -r /usr/lib64/nspluginwrapper 
 rm -r /usr/share/doc/nspluginwrapper*
 emerge -C nspluginwrapper
 

Amarok 2.4 Loses Track Statistics

Upgrading to Amarok 2.4.x will kill your old statistics. That is very disappointing, considering one of the primary reason for using this software is its statistic tracking. If you have already upgraded, this information is likely to not be of use; however, for those have yet to take the leap, read on as it might help you save your statistics.

Perhaps the best solution would be to enable writing the statistics to file. Unfortunately, I have not done this; however, if you are about to do an upgrade I would suggest this approach.

Settings > Configure Amarok > Collection > Write statistics to file

For those of us looking for something a bit more dynamic, the problem seems to be that with Amarok 2.4+ changes how it keeps track of your files. Previously, the database back-end stored the full path of the file. That has changed with the newer version of the program. Now it stores the disk path relative to the root(s).

For instance, I have a large music collection located on two disks: one is mounted at /home/pub/audio and the other at /home/pub/sound. Prior to 2.4 I noticed that the files were stored in the urls table with their full file path:

./home/pub/audio/100dbs/100dBs – 40 Minutes to Freedom, The Sublime Mix.mp3

After the upgrade, it is just:

./100dbs/100dBs – 40 Minutes to Freedom, The Sublime Mix.mp3

There is a deviceid column that keeps track which folder the file lives in.

I did find a reasonable workaround here, the short version is:

  • Backup your database
  • Launch amarok
  • Stop auto scan of music
  • Un-check “Watch folders for changes”
  • Restart amarok
  • Do a full collection scan
  • Quit amarok

This mostly worked for me; however, for some tracks I now had duplicates. Looking in the database I could see that the problem was that the old-style paths were still considered to be active and there were new-style paths too. To workaround this, I have written a script that looks for the all of the tracks that exist in the database that still begin with the old-style paths. For each of of these “old” stats, look for a corresponding “new” stat. If both are found, merge the stats. Finally, at the end of it all delete the records in the urls table that have a deviceid equal to negative one (i.e. old-style track records).

The python script for doing this is below. By default it is crippled, you must manually edit the script to contain your amarok database credentials as well as the location of your base location where your audio files live.
   1
   2
   3
   4
   5
   6
   7
   8
   9
  10
  11
  12
  13
  14
  15
  16
  17
  18
  19
  20
  21
  22
  23
  24
  25
  26
  27
  28
  29
  30
  31
  32
  33
  34
  35
  36
  37
  38
  39
  40
  41
  42
  43
  44
  45
  46
  47
  48
  49
  50
  51
  52
  53
  54
  55
  56
  57
  58
  59
  60
  61
  62
  63
  64
  65
  66
  67
  68
  69
  70
  71
  72
  73
  74
  75
  76
  77
  78
  79
  80
  81
  82
  83
  84
  85
  86
  87
  88
  89
  90
  91
  92
  93
  94
  95
  96
  97
  98
  99
 100
 101
 102
 103
 104
 105
 106
 107
 108
 109
 110
 111
 112
 113
 114
 115
 116
 117
 118
 119
 120
 121
 122
 123
 124
 125
 126
 127
 128
 129
 130
 131
 132
 133
 134
 135
 136
 137
 138
 139
 140
 141
 142
 143
 144
 145
 146
 147
 148
 149
 150
 151
 152
 153
 154
 155
 156
 157
 158
 159
 160
 161
 162
 163
 164
 165
 166
 167
 168
 169
 170
 171
 172
 173
 174
 175
 176
 177
 178
 179
 180
 181
 182
 183
 184
 185
 186
 187
 188
 189
 190
 191
 192
 193
 194
 195
 196
 197
 198
 199
 200
 201
 202
 203
 204
 205
 206
 207
 208
 209
 210
 211
 212
 213
 214
 215
 216
 217
 218
 219
 220
 221
 222
 223
 224
 225
 226
 227
 228
 229
 230
 231
 232
 233
 234
 235
 236
 237
 238
 239
 240
 241
 242
 243
 244
 245
 246
 247
 248
 249
 250
 251
 252
 253
 254
 255
 256
 257
 258
 259
 260
 261
 262
 263
 264
 265
 266
 267
 268
 269
 270
 271
 272
 273
 274
 275
 276
 277
 278
 279
 280
 281
 282
 283
 284
 285
 286
 287
 288
 289
 290
 291
 292
 293
 294
 295
 296
# * coding: utf-8 *
 import codecs, getopt, MySQLdb, os, random
 import string, subprocess, sys, time

 AMAROK_MUSIC_HOME = "/home/pub/audio"

 VERSION  = "0.0.1"
 DEBUG_FLAG = "false"
 VERBOSE_FLAG = "false"
 PRETEND_FLAG = "false"
 SCRIPT_NAME = "pyCorrectStats-" + VERSION + ".py"

 #######################################################################
 # 
 # Class for tweaking Amarok statistics
 #
 class CorrectAmarokStatistics:
   #######################################################################
   #
   # Update the statistics table
   #
   # For the given (Amarok) database connection, increase the playcount by
   # the given bump number
   #
   def correctAudioPlaycountByRemovingOldDuplicates(self, dbConnection):
     cursor = dbConnection.cursor()
     try:     
       # http://www.hashmysql.org/wiki/Select_rows_in_table_A_that_are_not_in_table_B
       #
       # How do I select all rows in table `a` that is not in table `b`:
       # SELECT a.id FROM a LEFT JOIN b ON a.id = b.aid WHERE b.aid IS NULL;
       #  

       # SELECT statistics.playcount,statistics.score,statistics.rating FROM statistics 
       # LEFT JOIN urls ON statistics.url = urls.id WHERE statistics.deleted = 0 AND 
       # urls.rpath = './home/pub/audio/100dbs/100dBs – 40 Minutes to Freedom, The Sublime Mix.mp3';

       findMismatchOfStatisticsAndUrl = """ SELECT urls.rpath,statistics.id FROM urls LEFT JOIN """ +\
       """statistics ON urls.id = statistics.url WHERE statistics.deleted = 0 AND urls.rpath REGEXP '^\.""" + \
       AMAROK_MUSIC_HOME + """'"""

       self.executeSelectStatement(cursor, findMismatchOfStatisticsAndUrl)
       verbose("Number of Mismatched Items : " + str(cursor.rowcount))

       # Look at the URL and see if the rpath does NOT start with ./home/pub.
       # If that is the case, the check if there is statistic for the same path
       # but does have ./home/pub/audio at the beginning.

       itemsToCheck =  cursor.fetchall()
       for item in itemsToCheck:
         oldRpath = str(item[0])
         oldRpath = str.replace(oldRpath, "'", "\\'.")
         oldStatsId = str(item[1])
         debug("Examining: " + oldRpath)

         newStylePath = str.replace(oldRpath, "." + AMAROK_MUSIC_HOME + "/", "./")

         findOldStatisticNumbers = """SELECT statistics.playcount,statistics.score,statistics.rating,urls.deviceid FROM statistics """ \
         + """LEFT JOIN urls ON statistics.url = urls.id WHERE statistics.deleted = 0 AND """ \
         + """urls.rpath = '""" + oldRpath + """'"""
         self.executeSelectStatement(cursor, findOldStatisticNumbers)
         if (0 < cursor.rowcount):
           verbose("Number of old stats for '" + oldRpath + "' : " + str(cursor.rowcount))
           oldStatsToMerge =  cursor.fetchall()
           oldPlaycount = oldStatsToMerge[0][0]
           oldScore = oldStatsToMerge[0][1]
           oldRating = oldStatsToMerge[0][2]
           oldDeviceId = oldStatsToMerge[0][3]

           findNewStatisticNumbers = """SELECT statistics.playcount,statistics.score,statistics.rating,statistics.id FROM statistics """ \
           + """LEFT JOIN urls ON statistics.url = urls.id WHERE statistics.deleted = 0 AND """ \
           + """urls.rpath = '""" + newStylePath + """'"""
           self.executeSelectStatement(cursor, findNewStatisticNumbers)

           if (1 == cursor.rowcount):
             verbose("Number of new stats for '" + newStylePath + "' : " + str(cursor.rowcount))
             newStatsToMerge =  cursor.fetchall()
             newPlaycount = newStatsToMerge[0][0]
             newScore = newStatsToMerge[0][1]
             newRating = newStatsToMerge[0][2]
             newId = newStatsToMerge[0][3]
             verbose("Number of new stats for '" + newStylePath + "' : " + str(cursor.rowcount))

             #
             # 1) Update the old stat to be deleted
             #
             markOldStatAsDeleted = """UPDATE statistics SET statistics.deleted = 1 WHERE statistics.id = """ + str(oldStatsId)
             self.executeUpdateStatement(cursor, markOldStatAsDeleted)

             #
             # 2) Merge the statistics into the new stat
             #
             finalPlaycount = oldPlaycount + newPlaycount
             finalScore = 10
             if (oldScore != None):
               finalScore = oldScore
             if (newScore != None):
               finalScore = (finalScore + newScore)/2
             finalRating = (oldRating + newRating)/2

             mergeStatsIntoNewRecord = """UPDATE statistics SET statistics.playcount = """ + str(finalPlaycount) 
             mergeStatsIntoNewRecord = mergeStatsIntoNewRecord + """, statistics.score = """ + str(finalScore)
             mergeStatsIntoNewRecord = mergeStatsIntoNewRecord + """, statistics.rating = """ + str(finalRating)
             mergeStatsIntoNewRecord = mergeStatsIntoNewRecord + """ WHERE statistics.id = """ + str(newId) + """;"""

             self.executeUpdateStatement(cursor, mergeStatsIntoNewRecord)

       # outside of loop
       #
       # 3) delete if deviceid equals zero
       #
       deleteDeviceIdNegativeOne = """DELETE urls FROM urls LEFT JOIN statistics ON urls.id = statistics.url WHERE urls.deviceid = -1"""
       self.executeDeleteStatement(cursor, deleteDeviceIdNegativeOne)
       verbose("number of deleted tracks: " + str(cursor.rowcount))

     except Exception as detail:
       warning(str(detail))
     finally:
       debug("Closing the cursor")
       cursor.close()
     return

   #######################################################################
   #
   # Run a SQL DELETE statement
   #
   def executeDeleteStatement(self, cursor, statement):
     debug("Running DELETE = '" + statement +"'")
     if PRETEND_FLAG == 'false':
       cursor.execute (statement)
       debug("Executed DELETE = '" + statement +"'")
     debug("Number of rows found from DELETE: " + str(cursor.rowcount)) 

 #######################################################################

   #######################################################################
   #
   # Run a SQL INSERT statement
   #
   def executeInsertStatement(self, cursor, statement):
     debug("Running INSERT = '" + statement +"'")
     if PRETEND_FLAG == 'false':
       cursor.execute (statement)
       debug("Executed INSERT = '" + statement +"'")
     debug("Number of rows found from INSERT: " + str(cursor.rowcount))
     return

   #######################################################################
   #
   # Run a SQL SELECT statement
   #
   def executeSelectStatement(self, cursor, statement):
     debug("Running SELECT = '" + statement +"'") 
     cursor.execute (statement)
     debug("Executed SELECT = '" + statement +"'") 
     debug("Number of rows found from SELECT: " + str(cursor.rowcount))
     return

   #######################################################################
   #
   # Run a SQL UPDATE statement
   #
   def executeUpdateStatement(self, cursor, statement):
     debug("Running UPDATE = '" + statement +"'")
     if PRETEND_FLAG == 'false':
       cursor.execute (statement)
       debug("Executed UPDATE = '" + statement +"'")
     debug("Number of rows found from UPDATE: " + str(cursor.rowcount)) 

 #######################################################################
 #
 # Establish a connection to the Amarok database
 #
 def connectToDatabase(machineName, databaseName, username, password):
   debug("\n\rEntering connectToDatabase()")
   debug("  hostname= " + machineName)
   debug("  databaseName = " + databaseName)
   debug("  username = " + username)
   debug("  password = <Ssssh>")
   return MySQLdb.connect (host = machineName,
                            user = username,
                            passwd = password,
                            db = databaseName) 

 #######################################################################
 #
 # Prints debugging messages (if the debugging flag has been set)
 #
 def debug(message):
   if DEBUG_FLAG == "true":
     print(message)

 #######################################################################
 #
 # Prints verbose messages (if the debugging flag has been set)
 #
 def verbose(message):
   if VERBOSE_FLAG  "true" or DEBUG_FLAG  "true":
     print(message)

 def correctAmarokStats(connection):
   bump = CorrectAmarokStatistics()
   bump.correctAudioPlaycountByRemovingOldDuplicates(connection)

 #######################################################################
 #
 # Prints warning messages to STDERR
 #
 def warning(message):
   print >> sys.stderr, "WARNING : " + message

 #######################################################################
 #
 # Print Usage Information
 #
 def usage():
   print("")
   print("Usage information:")
   print("")
   print("Before proceeding, open the script and replace the variable AMAROK_MUSIC_HOME ")
   print("with your local path on disk.")
   print("")
   print("Also, update the MySQL database connection information.")
   print("")
   print("\t" + SCRIPT_NAME)
   print("")

 #######################################################################
 #
 # Start of the 'main()' function
 # 
 def main():
   #######################################################################
   #
   # Read Command-Line
   #
   global SCRIPT_NAME
   SCRIPT_NAME = sys.argv[0]
   try:
     opts, args = getopt.getopt(sys.argv[1:], "hdvp", \
                  ["help", "debug", "verbose", "pretend"])
   except getopt.GetoptError, err:
     # print help information and exit:
     warning(str(err))
     usage()
     sys.exit(2)
   #######################################################################
   #
   # Parse Command-Line Options and Arguments
   # 
   playCountBump = 0
   for o, a in opts:
     if o in ("h", "-help"):
       usage()
       sys.exit()
     elif o in ("d", "-debug"):
       global DEBUG_FLAG
       DEBUG_FLAG = "true"
     elif o in ("v", "-verbose"):
       global VERBOSE_FLAG
       VERBOSE_FLAG = "true"
     elif o in ("p", "-pretend"):
       global PRETEND_FLAG
       PRETEND_FLAG = "true"
     else:
       assert False, "unhandled option"

   #######################################################################
   #
   # Sanity checks 
   #

   #
   # Remove the following lines before using
   #
   usage()
   sys.exit(0)

   debug(str(args))
   if len(args) != 0:
     warning("No non-switch command-line arguments allowed")
     usage()
   else:
     debug("\n\rConnecting to database")
     connection = connectToDatabase("localhost", "amarok_toy", "amarokuser", "amarokpasswd")

     try:
       correctAmarokStats(connection)
     finally:
       debug("Closing connection to database")
       connection.close()

 if name == "main":
     main()
 

Download the Script:

KDE 4.6 and Sleep/Hibernate

Recently I upgraded from KDE 4.4 to 4.6. It was fairly smooth aside from some difficulties with amarok (that will be another post), but I did lose the ability to sleep/hibernate. Typically, this problem is due to lack of proper kernel configuration; however, not in my case.

The problem is best described in this Gentoo Forum posting. The short version is that there is a permission problem. In my case, I needed to add/amend the org.freedesktop.upower.policy file in /usr/share/polkit-1/actions/.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
<?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE policyconfig PUBLIC
  "-//freedesktop//DTD PolicyKit Policy Configuration 1.0//EN"
  "http://www.freedesktop.org/standards/PolicyKit/1.0/policyconfig.dtd">
 <policyconfig>
   <vendor>The UPower Project</vendor>
   <vendor_url>http://upower.freedesktop.org/</vendor_url>
   <icon_name>system-suspend</icon_name>

   <action id="org.freedesktop.upower.suspend">
     <description>Suspend the system</description>
     <message>Authentication is required to suspend the system</message>
     <defaults>
       <allow_inactive>no</allow_inactive>
       <allow_active>yes</allow_active>
     </defaults>
   </action>

   <action id="org.freedesktop.upower.hibernate">
     <description>Hibernate the system</description>
     <message>Authentication is required to hibernate the system</message>
     <defaults>
       <allow_inactive>no</allow_inactive>
       <allow_active>yes</allow_active>
     </defaults>
   </action>
 </policyconfig>
 

Embedding Images in MP3 Files: version 0.0.4

This is the fourth revision of a python script for embedding artwork into MP3 (and other) audio files. This script has been verified to run on Linux, Mac OS X, and Windows. This edition fixes a couple of minor bugs introduced in version 0.0.3.

   1
   2
   3
   4
   5
   6
   7
   8
   9
  10
  11
  12
  13
  14
  15
  16
  17
  18
  19
  20
  21
  22
  23
  24
  25
  26
  27
  28
  29
  30
  31
  32
  33
  34
  35
  36
  37
  38
  39
  40
  41
  42
  43
  44
  45
  46
  47
  48
  49
  50
  51
  52
  53
  54
  55
  56
  57
  58
  59
  60
  61
  62
  63
  64
  65
  66
  67
  68
  69
  70
  71
  72
  73
  74
  75
  76
  77
  78
  79
  80
  81
  82
  83
  84
  85
  86
  87
  88
  89
  90
  91
  92
  93
  94
  95
  96
  97
  98
  99
 100
 101
 102
 103
 104
 105
 106
 107
 108
 109
 110
 111
 112
 113
 114
 115
 116
 117
 118
 119
 120
 121
 122
 123
 124
 125
 126
 127
 128
 129
 130
 131
 132
 133
 134
 135
 136
 137
 138
 139
 140
 141
 142
 143
 144
 145
 146
 147
 148
 149
 150
 151
 152
 153
 154
 155
 156
 157
 158
 159
 160
 161
 162
 163
 164
 165
 166
 167
 168
 169
 170
 171
 172
 173
 174
 175
 176
 177
 178
 179
 180
 181
 182
 183
 184
 185
 186
 187
 188
 189
 190
 191
 192
 193
 194
 195
 196
 197
 198
 199
 200
 201
 202
 203
 204
 205
 206
 207
 208
 209
 210
 211
 212
 213
 214
 215
 216
 217
 218
 219
 220
 221
 222
 223
 224
 225
 226
 227
 228
 229
 230
 231
 232
 233
 234
 235
 236
 237
 238
 239
 240
 241
 242
 243
 244
 245
 246
 247
 248
 249
 250
 251
 252
 253
 254
 255
 256
 257
 258
 259
 260
 261
 262
 263
 264
 265
 266
 267
 268
 269
 270
 271
 272
 273
 274
 275
 276
 277
 278
 279
 280
 281
 282
 283
 284
 285
 286
 287
 288
 289
 290
 291
 292
 293
 294
 295
 296
 297
 298
 299
 300
 301
 302
 303
 304
 305
 306
 307
 308
 309
 310
 311
 312
 313
 314
 315
 316
 317
 318
 319
 320
 321
 322
 323
 324
 325
 326
 327
 328
 329
 330
 331
 332
 333
 334
 335
 336
 337
 338
 339
 340
 341
 342
 343
 344
 345
 346
 347
 348
 349
 350
 351
 352
 353
 354
 355
 356
 357
 358
 359
 360
 361
 362
 363
 364
 365
 366
 367
 368
 369
 370
 371
 372
 373
 374
 375
 376
 377
 378
 379
 380
 381
 382
 383
 384
 385
 386
 387
 388
 389
 390
 391
 392
 393
 394
 395
 396
 397
 398
 399
 400
 401
 402
 403
 404
 405
 406
 407
 408
 409
 410
 411
 # * coding: utf-8 *
 import getopt, os, platform, subprocess, sys

 version  = "0.0.4"
 imageExt = [[".jpg", "image/jpeg"], [".png", "image/png"], [".gif", "image/gif"]]
 USE_MUTAGEN = "false"
 USE_FIRST_IMAGE_FOUND = "false"
 MUTAGEN_MODULE = ""
 CMD_CONVERT = "convert"
 CMD_EYED3 = "eyeD3"
 CMD_ID3 = "id3v2"
 DEBUG_FLAG = "false"
 DEFAULT_RESOLUTION = "600×600"
 coverJpg = "cover-embed-" + DEFAULT_RESOLUTION + ".jpg"
 coverPng = "cover-embed-" + DEFAULT_RESOLUTION + ".png"

 #######################################################################
 #
 # Method called to walk the directory tree
 #
 # args: extension to find
 # dirname: name of current directory
 # list of files in the current directory
 #
 def find(arg, dirname, names):
   foundMp3 = "false"
   for item in names:
     pathname = os.path.join(dirname, item)
     pathname = os.path.normpath(pathname)
     fileExtension = os.path.splitext( item )
     if os.path.isfile(pathname) and fileExtension[1].lower() == arg:
       debug("Found " + arg + " in " + dirname)
       foundMp3 = "true"
       break

   if foundMp3 == "false":
     debug("No "+ arg +"s found in directory " + dirname)
     return

   # (1) Get an image to embed
   fullCoverJpg = findImage(dirname, names)

   if fullCoverJpg is None or len(fullCoverJpg) == 0:
     debug("No acceptable image found for " + dirname)
     return;
   
   # 2) Loop through contents and determine
   # if it is a file or a directory. Directories
   # can be skipped. Whereas for files, we'll
   # want to make a callback to embed the images.
   badAudioFiles = []
   for item in names:
     pathname = os.getcwd()
     pathname = os.path.join(pathname, dirname)
     pathname = os.path.join(pathname, item)
     pathname = os.path.normpath(pathname)
     if os.path.isfile(pathname) and pathname.lower().find(arg)>0:
       embedReturnCode = embedImage(pathname, fullCoverJpg)
       if 0 < embedReturnCode:
         badAudioFiles.append(pathname)
   if 0 < len(badAudioFiles):
     print "Errors occurred embedding images in the following files:"
     sys.stdout.write(" * ")
     print "\n\r * ".join(badAudioFiles)

 #######################################################################
 #
 # Given a directory, attempt to find a suitable cover image
 #
 def findImage(directory, filenames):
   imageFileToUse = ""
   # 1) Look in directory for if there is an
   # album cover, leave if not
   if coverJpg in filenames:
     debug("Found JPEG cover image, no conversion necessary")
     fullCoverJpg = os.path.join(directory, coverJpg)
     fullCoverJpg = os.path.normpath(fullCoverJpg)
     imageFileToUse = [fullCoverJpg, "image/jpeg"]
   elif coverPng in filenames:
     # No JPEG image found, but a legally named PNG exists
     imageFileToUse = convertImage(directory, coverPng, coverJpg)
   else:
     desiredImage = guessImageFile(directory, filenames);
     # found an acceptable image (based on extension)
     # if it is already a JPG, just make a copy. Otherwise do a
     # do a conversion
     if desiredImage  "":
       print >> sys.stderr, "no image found in " + directory
     elif USE_FIRST_IMAGE_FOUND  "true":
       imageFileToUse = desiredImage
     elif len(desiredImage[0]) > 0:
       imageFileToUse = convertImage(directory, desiredImage[0], coverJpg)
     # by virtue of making it out of the loop, we have admitted that
     # no acceptable image was present.
   return imageFileToUse;

 #######################################################################
 #
 # Simple heuristic for determining the correct cover art file.
 #
 def guessImageFile(directory, filenames):
   # There is no well-defined image for this directory.
   # Attempt to guestimate a reasonable replacement.  Look
   # through what is available & sort those 
   potentialImages = []
   for aFile in filenames:
     for extension,mime in imageExt:
       fileExtension = os.path.splitext( aFile )
       if fileExtension[1].lower()   extension:
 	debug("Looking at using file: " + aFile + " | Mime Type: " + mime)
 	fullpath = os.getcwd()
         fullpath = os.path.join(fullpath, directory)
         fullpath = os.path.join(fullpath, aFile)
         fullpath = os.path.normpath(fullpath)
         potentialImages.append([fullpath, mime])
       
   desiredImage = ""
   for currentFile, currentMime in potentialImages:
     if len(desiredImage) > 0:
       break;
     goodKeywords = [ "front", "large", "big"]
     for keyword in goodKeywords:
       if currentFile.lower().find(keyword) > 0:
         desiredImage = [currentFile, currentMime]
         debug("found image: " + currentFile)
         break;
 	
   # The first heuristic did not work, just pick the first image
   if desiredImage is None or desiredImage  "":
     if len(potentialImages) > 0:
       debug("No image found, defaulting to " + potentialImages[0][0])
       desiredImage = potentialImages[0];

   return desiredImage

 #######################################################################
 #
 # Linux command-line approach for converting/shrinking an image file.
 #
 # directory: directory where the file lives
 # foundImage: fullpath to the the desired image
 # desiredFilename: name of the output file
 #
 def convertImage(directory, foundImage, desiredFilename):
   # With imagemagick version 6.4.8.3, the command
   # to convert the image is similar to:
   # convert cover.png -geometry 600×600 cover.jpg
   fullCoverJpg = os.path.join(directory, desiredFilename)
   fullCoverJpg = os.path.normpath(fullCoverJpg)
   cmd = [CMD_CONVERT, foundImage, "-geometry", DEFAULT_RESOLUTION , fullCoverJpg]
   shellCommandWrapper(cmd)
   return [fullCoverJpg, "image/jpeg"]

 #######################################################################
 #
 # Wrapper function to pick the correct means of embedding the 
 # album artwork into the audio file.
 #
 def embedImage(audiofile, image):
   addReturnCode = 0
   
   if USE_MUTAGEN:
     addReturnCode = embedImageViaMutagen(audiofile, image)
   else:
     addReturnCode = embedImageViaLinuxCommandLine(audiofile, image)

   return addReturnCode

 #######################################################################
 #
 # Use the "v2" approach of Mutagen
 #
 def embedImageViaMutagen(audiofile, image):
   returnCode = 0
   exec "from mutagen.mp3 import MP3"
   exec "from mutagen.id3 import ID3, APIC, error"
   audioTags = ID3(audiofile)
   try:
     debug("Adding " + image[0] + " to the file " + audiofile )
     audioTags.add(APIC(encoding=3, mime=image[1], type=3, desc=u'Cover',data=open(image[0]).read()))
     audioTags.save()
   except error:
     returnCode = 1

   return returnCode

 #######################################################################
 #
 # Use the "v1" approach of several command-line tools
 #
 def embedImageViaLinuxCommandLine(audiofile, image):
   removeReturnCode = 0
   addReturnCode = 0
   removeReturnCode  += zeroBpm(audiofile);

   # (1) Remove all existing images
   removeReturnCode += removeOtherImage(audiofile);
   removeReturnCode += removeOtherImage(audiofile);
   removeReturnCode += removeFrontImage(audiofile);
   removeReturnCode += removeFrontImage(audiofile);

   # (2) remove cruft
   removeReturnCode += removeCruft(audiofile);

   # (3) check if valid ID3 tag
   removeReturnCode += checkValidId3Tag(audiofile);

   # (4) add front image
   addReturnCode += addFrontImage(audiofile, image[0]);
   
   if 0 < removeReturnCode:
     debug("——————————————————————————————————")
     debug("Errors occured when removing/cleaning ID3 tag for: ")
     debug(audiofile)
     debug("This behavior is unexpected, but may not impact embedding the image.")
     debug("——————————————————————————————————")
     
   if 0 < addReturnCode:
     debug("——————————————————————————————————")
     debug("The unable to embed image in: ")
     debug(audiofile)
     debug("——————————————————————————————————")
   
   return addReturnCode

 #######################################################################
 #
 # Linux command-line wrapper for adding the front cover artwork
 #
 def addFrontImage(theFile, imageFile):
   imagePathArg = "—add-image=" + imageFile + ":FRONT_COVER" 
   cmd = [CMD_EYED3, "—no-color", imagePathArg, theFile]
   return shellCommandWrapper(cmd)

 #######################################################################
 #
 # Linux command-line wrapper for converting to the right ID3 version
 #
 def checkValidId3Tag(theFile):
   cmd = [CMD_EYED3, "—no-color", "—to-v2.4", theFile]
   return shellCommandWrapper(cmd)

 #######################################################################
 #
 # Linux command-line wrapper hack for dealing with floating-point BPMs
 #
 def zeroBpm(theFile):
   cmd = [CMD_EYED3, "—bpm=90", theFile]
   return shellCommandWrapper(cmd)

 #######################################################################
 #
 # Linux command-line wrapper for removing the front cover artwork
 #
 def removeFrontImage(theFile):
   cmd = [CMD_EYED3, "—no-color", "—add-image=:FRONT_COVER", theFile]
   return shellCommandWrapper(cmd)

 #######################################################################
 #
 # Linux command-line wrapper for removing the other artwork
 #
 def removeOtherImage(theFile):
   cmd = [CMD_EYED3, "—no-color", "—add-image=:OTHER", theFile]
   return shellCommandWrapper(cmd)

 #######################################################################
 #
 # Linux command-line wrapper for removing cruft
 #  * Mainly needed for older ID3 editors, may no longer be an issue
 #    on newer (2010+) installs
 #
 def removeCruft(theFile):
   cmd = [CMD_ID3, "-APIC", "", theFile]
   return shellCommandWrapper(cmd)

 #######################################################################
 #
 # Simple environment sanity check
 #
 def checkEnvironment():
   #
   # try to dynamically load mutagen
   #
   try:
     global MUTAGEN_MODULE 
     MUTAGEN_MODULE = import("mutagen")
     debug(MUTAGEN_MODULE)
     USE_MUTAGEN = "true"
     debug("Able to dynamically load mutagen")
   except:
     debug("Unable to dynamically load mutagen")
     if "Linux" <> platform.system():
       print >> sys.stderr, "*********************************************************"
       print >> sys.stderr, "***            Environment Check Failed!              ***"
       print >> sys.stderr, "***                                                   ***"
       print >> sys.stderr, "               mutagen must be installed"
       print >> sys.stderr, "***                                                   ***"
       print >> sys.stderr, "            http://code.google.com/p/mutagen/"
       print >> sys.stderr, "***                                                   ***"
       print >> sys.stderr, "*********************************************************"
       print
       sys.exit(1)
     #
     # fallback 'raw' method, look for the appropriate command-line utilities
     #
     checkEnvironmentHelper([CMD_EYED3, "—help"])
     checkEnvironmentHelper([CMD_ID3])
   if "Linux"  platform.system():
     checkEnvironmentHelper([CMD_CONVERT])
   else:
     global USE_FIRST_IMAGE_FOUND
     USE_FIRST_IMAGE_FOUND = "true"
   
 #######################################################################
 #
 # Helper method for displaying errors about some simple command-line
 # environment checks (for Linux)
 #
 def checkEnvironmentHelper(command):  
   if 0 < shellCommandWrapper(command):
     print >> sys.stderr, "*********************************************************"
     print >> sys.stderr, "***            Environment Check Failed!              ***"
     print >> sys.stderr, " Unable to locate '" + command[0] + "' in the PATH"
     print >> sys.stderr, "*********************************************************"
     print
  
 #######################################################################
 #
 # Prints debugging messages (if the debugging flag has been set)
 #
 def debug(message):
   if DEBUG_FLAG  "true":
     print message

 #######################################################################
 #
 # Small wrapper method for executing Linux Shell Commands
 #
 def shellCommandWrapper(command):
   FNULL = open(os.devnull, 'w')
   process = subprocess.Popen(command, shell=False, bufsize=1, \
             stdin=None, stdout=FNULL, stderr=FNULL)
   process.wait()
   return process.returncode
   
 #######################################################################
 #
 # Print Usage Information
 #
 def usage():
   print
   print "This script takes one or more arguments that are expected to be a "     + \
         "a single directory or set of directories. For each argument supplied, " + \
         "the script will recurse into the directory and look for an image file. If a " + \
         "file is found, it will be converted to the default resolution (" + DEFAULT_RESOLUTION + \
         "). The script will then look for any MP3s in the directory. If found, the " + \
         "image will be embedded into each file (overwriting an existing embedded image)."

 #######################################################################
 #
 #
 # Start of the 'main()' function
 # 
 def main():
   #######################################################################
   #
   # Read Command-Line
   #
   try:
     opts, args = getopt.getopt(sys.argv[1:], "hd", ["help", "debug", \
                                "use-first-image", "use-mutagen"])
   except getopt.GetoptError, err:
     # print help information and exit:
     print >> sys.stderr, err
     usage()
     sys.exit(2)
   #######################################################################
   #
   # Parse Command-Line Options and Arguments
   # 
   for o, a in opts:
     if o in ("h", "-help"):
       usage()
       sys.exit()
     elif o in ("d", "-debug"):
       global DEBUG_FLAG
       DEBUG_FLAG = "true"
     elif o  "--use-first-image":
       global USE_FIRST_IMAGE_FOUND
       USE_FIRST_IMAGE_FOUND = "true"
     elif o  "—use-mutagen":
       global USE_MUTAGEN
       USE_MUTAGEN = "true"
     else:
       assert False, "unhandled option"
   #######################################################################
   #
   # Now that all the processing is out of the way, walk the directory
   # structure and embed artwork into any discovered MP3 files.
   #
   if len(args) == 0:
     print >> sys.stderr, "Missing command-line arguments"
     usage()
   else:
     checkEnvironment()
     for directory in args:
       os.path.walk(directory, find, ".mp3")

 if name == "main":
     main()
 

Download the Script:

Python Script for Bumping Playcount in Amarok2: version 0.0.1

The primary audio-player I use at home is Amarok. At work, I always listen to music. For no particular reason, I like to keep track of how much I listen to certain albums.

Unfortunately, Amarok does not provide a means of setting or updating the playcount directory. As I chose a MySQL back-end, luckily it wasn’t too hard to write a script to accomplish this task.

Backup

Before getting started, back-up your Amarok database.

What the script does

This script increases the playcount for a set of files in a directory. The script does this in the following steps:

  • For each file in the directory, check if it is a audio file (mp3, flac, ogg, m4a).
  • Find the id of the file’s url by looking in the urls table.
  • Next look for an entry in the statistics table.
  • If there is an entry, increase the playcount by the given number and update the date of the last access to now.
  • If there is not an entry, set the playcount to the given number and set both the create and access date to now. Additionally, set the score and rating to 0 (feel free to adjust these your desired hard-code number).

Prerequisites

This script was developed against the following:

  • Amarok: 2.3.1
  • MySQL: 5.1.51
  • Python: 2.6.5

Download the Script

Usage

Before using the script, you need to edit the script. With your favorite text editor, open the script and change the creditials for the Amarok database name, username, and password credentials. Once this is done, you are ready to update your playcount.

If you have not already done so, be sure to back-up your Amarok database.

To increase the playcount of all files in the directory, call the script with a given playcount number and the full path to the directory you wish to update. For example, the command below would increase the playcount of the No Sleep For A Day EP by 3.

python pyAmarokBump-0.0.1.py --playcount-bump=3 /home/pub/audio/Blu/NoSleepForADayEP/
 

The Script

   1
   2
   3
   4
   5
   6
   7
   8
   9
  10
  11
  12
  13
  14
  15
  16
  17
  18
  19
  20
  21
  22
  23
  24
  25
  26
  27
  28
  29
  30
  31
  32
  33
  34
  35
  36
  37
  38
  39
  40
  41
  42
  43
  44
  45
  46
  47
  48
  49
  50
  51
  52
  53
  54
  55
  56
  57
  58
  59
  60
  61
  62
  63
  64
  65
  66
  67
  68
  69
  70
  71
  72
  73
  74
  75
  76
  77
  78
  79
  80
  81
  82
  83
  84
  85
  86
  87
  88
  89
  90
  91
  92
  93
  94
  95
  96
  97
  98
  99
 100
 101
 102
 103
 104
 105
 106
 107
 108
 109
 110
 111
 112
 113
 114
 115
 116
 117
 118
 119
 120
 121
 122
 123
 124
 125
 126
 127
 128
 129
 130
 131
 132
 133
 134
 135
 136
 137
 138
 139
 140
 141
 142
 143
 144
 145
 146
 147
 148
 149
 150
 151
 152
 153
 154
 155
 156
 157
 158
 159
 160
 161
 162
 163
 164
 165
 166
 167
 168
 169
 170
 171
 172
 173
 174
 175
 176
 177
 178
 179
 180
 181
 182
 183
 184
 185
 186
 187
 188
 189
 190
 191
 192
 193
 194
 195
 196
 197
 198
 199
 200
 201
 202
 203
 204
 205
 206
 207
 208
 209
 210
 211
 212
 213
 214
 215
 216
 217
 218
 219
 220
 221
 222
 223
 224
 225
 226
 227
 228
 229
 230
 231
 232
 233
 234
 235
 236
 237
 238
 239
 240
 241
 242
 # * coding: utf-8 *
 import getopt, MySQLdb, os, string, subprocess, sys, time, urllib

 VERSION  = "0.0.1"
 DEBUG_FLAG = "false"
 PRETEND_FLAG = "false"
 SCRIPT_NAME = "pyAmBump-" + VERSION + ".py"

 #######################################################################
 #
 # args: extention to find
 # dirname: name of current directory
 # list of files in the current directory
 #
 def find(bump, dirname, names):
   debug("\n\rEntering find()")
   debug("  bump = " + str(bump))
   debug("  dirname = " + dirname)
   debug("  names = " + str(names))
   foundAudioFiles = "false"
   for item in names:
     fullFilePath = os.path.abspath(os.getcwd())
     fullFilePath = os.path.join(fullFilePath, dirname)
     fullFilePath = os.path.join(fullFilePath, item)

     if os.path.isfile(fullFilePath):
       result = findHelper(dirname, item, fullFilePath, bump)
       if result == "true":
         foundAudioFiles = "true"

   if foundAudioFiles == "false":
     debug("No audiofiles found in directory: " + dirname)
     return

 #######################################################################
 #
 # For the given (Amarok) database connection, increase the playcount by
 # the given bump number
 #
 def findHelper(dirname, item, fullFilePath, bump):
   foundAudioFiles = "false"
   for audioType in [".mp3", ".flac", ".ogg", ".m4a"]:
     fileExtension = os.path.splitext( fullFilePath )
     if fileExtension[1].lower() == audioType:
       debug("\n\rFound " + audioType + " in " + dirname + ": " + item)
       foundAudioFiles = "true"
       bumpAudioPlaycount(bump[0], fullFilePath, bump[1])
   return foundAudioFiles

 #######################################################################
 #
 # Update the statistics table
 #
 # For the given (Amarok) database connection, increase the playcount by
 # the given bump number
 #
 def bumpAudioPlaycount(dbConnection, fullFilePath, bump):
   debug("\n\ra) find id from the url/path")
   cursor = dbConnection.cursor()
   try:
     findUrlIdStatement = """
     SELECT id FROM urls WHERE 
     urls.rpath = ".""" + fullFilePath + """";"""
     executeSelectStatement(cursor, findUrlIdStatement)
     debug("curor.rowcount : " + str(cursor.rowcount))
     createAndAccessDate = str(int(time.time()))
     
     if cursor.rowcount != 1:
       warning("No entry in url table for " + fullFilePath)
     else:
       debug("\n\rb) update/add current playcount")
       findUrlIdResult =  cursor.fetchall()
       urlId = str(findUrlIdResult[0][0])   
       
       # find the appropriate entry in the statistics table and updatePlaycountStatement
       findStatisticsIdStatement = """SELECT id FROM statistics WHERE statistics.url = """ + urlId + """;"""
       executeSelectStatement(cursor, findStatisticsIdStatement)

       if cursor.rowcount == 0:
         debug("\n\rc – i) create playcount record")   
         debug("Inserted play count will be " + str(int(bump)))
         insertPlaycountStatement = """INSERT INTO statistics (url,createdate,accessdate,score,rating,playcount) VALUES(""" + \
                                    urlId + """, """ + createAndAccessDate + """, """ + createAndAccessDate + \
                                    """, 0, 0, """ + str(bump)  + """);""" 
         findPlayCountResult = executeInsertStatement(cursor, insertPlaycountStatement)
       
       else:
         debug("\n\rc – ii) update/add current playcount")
         findPlaycountResult =  cursor.fetchall()
         dbPlayCount = findUrlIdResult[0][0]
         debug("Current playcount is " + str(dbPlayCount))
         debug("Updated play count will be " + str(dbPlayCount + int(bump)))
         updatePlaycountStatement = """UPDATE statistics SET statistics.playcount = statistics.playcount + """ \
                                    + str(bump) 
         updatePlaycountStatement = updatePlaycountStatement + """, statistics.accessdate = """ \
                                    + createAndAccessDate
         updatePlaycountStatement = updatePlaycountStatement + """ WHERE statistics.url = """ \
                                    + urlId + """;"""
         findPlayCountResult = executeUpdateStatement(cursor, updatePlaycountStatement)

   except Exception as detail:
     warning(str(detail))
     debug("Error while updating record for : " + fullFilePath)
   finally:
     debug("Closing the cursor")
     cursor.close()

 #######################################################################
 #
 # Run a SQL INSERT statement
 #
 def executeInsertStatement(cursor, statement):
   debug("Running INSERT = '" + statement +"'")
   if PRETEND_FLAG == 'false':
     cursor.execute (statement)
     debug("Executed INSERT = '" + statement +"'") 
   debug("Number of rows found from INSERT: " + str(cursor.rowcount))

 #######################################################################
 #
 # Run a SQL SELECT statement
 #
 def executeSelectStatement(cursor, statement):
   debug("Running SELECT = '" + statement +"'") 
   cursor.execute (statement)
   debug("Executed SELECT = '" + statement +"'") 
   debug("Number of rows found from SELECT: " + str(cursor.rowcount))

 #######################################################################
 #
 # Run a SQL UPDATE statement
 #
 def executeUpdateStatement(cursor, statement):
   debug("Running UPDATE = '" + statement +"'")
   if PRETEND_FLAG == 'false':
     cursor.execute (statement)
     debug("Executed UPDATE = '" + statement +"'")
   debug("Number of rows found from UPDATE: " + str(cursor.rowcount))

 #######################################################################
 #
 # Establish a connection to the Amarok database
 #
 def connectToDatabase(machineName, databaseName, username, password):
   debug("\n\rEntering connectToDatabase()")
   debug("  hostname= " + machineName)
   debug("  databaseName = " + databaseName)
   debug("  username = " + username)
   debug("  password = <Ssssh>")
   return MySQLdb.connect (host = machineName,
                            user = username,
                            passwd = password,
                            db = databaseName) 

 #######################################################################
 #
 # Prints debugging messages (if the debugging flag has been set)
 #
 def debug(message):
   if DEBUG_FLAG == "true":
     print message
     
 #######################################################################
 #
 # Prints warning messages to STDERR
 #
 def warning(message):
   print >> sys.stderr, "WARNING : " + message

 #######################################################################
 #
 # Print Usage Information
 #
 def usage():
   print
   print "Usage information"
   print
   print "To increase the playcount of a directory of audio files, "
   print "execute the following:"
   print
   print "\t" + SCRIPT_NAME + "  —playcount-bump=<number> [Path to File(s)]"  

 #######################################################################
 #
 # Start of the 'main()' function
 # 
 def main():
   #######################################################################
   #
   # Read Command-Line
   #
   global SCRIPT_NAME
   SCRIPT_NAME = sys.argv[0]
   try:
     opts, args = getopt.getopt(sys.argv[1:], "hdp", ["help", "debug", "pretend", "playcount-bump="])
   except getopt.GetoptError, err:
     # print help information and exit:
     print >> sys.stderr, err
     usage()
     sys.exit(2)
   #######################################################################
   #
   # Parse Command-Line Options and Arguments
   # 
   playCountBump = 0
   for o, a in opts:
     if o in ("h", "-help"):
       usage()
       sys.exit()
     elif o in ("d", "-debug"):
       global DEBUG_FLAG
       DEBUG_FLAG = "true"
     elif o in ("p", "-pretend"):
       global PRETEND_FLAG
       PRETEND_FLAG = "true"
     elif o  "--playcount-bump":
       playCountBump = a
     else:
       assert False, "unhandled option"
   #######################################################################
   #
   # Sanity checks 
   #
   if len(args)  0:
     print >> sys.stderr, "Missing command-line arguments"
     usage()
   elif playCountBump < 1:
     print >> sys.stderr, "A positive playcount bump number must be specified"
     usage()
   else:
     debug("Connecting to database")
     connection = connectToDatabase("localhost", "yourAmarokDatabase ", "yourAmarokUsername", "yourAmarokPasswd")
     try:
       bumpInfo =  [ connection, playCountBump]
       os.path.walk(string.join(args), find, bumpInfo)
     finally:
       debug("Closing connection to database")
       connection.close()

 if name == "main":
     main()
 

Backup/Restore a (Amarok) MySQL Database

Backing-up and restoring an Amarok MySQL database is no different than any other MySQL database.

How to Backup a MySQL Database

The mysqldump command is used to export the database as a series of SQL statements to a text file.

$ mysqldump -u [username] -p [password] [databasename] > [dump_file.sql]
  • [username] – database username
  • [password] – database password
  • [databasename] – database name
  • [backupfile.sql] – the file to which the backup should be written.

How to Create a new MySQL Database

Creating a MySQL database is a standard procedure

  • http://www.gentoo-wiki.info/Amarok
  • http://www.debuntu.org/how-to-create-a-mysql-database-and-set-privileges-to-a-user
$ mysql -p -u root
>create database amarokDatabase;
>use amarokDatabase;
>grant all on amarokDatabase.* to amarokuser@localhost identified by '<your_password_here>';
>flush privileges;
  • amarokDatabase is the name of your database
  • amarokuser is the name of the database user account
  • <your_password_here> is the password for the user account

How to Restore a MySQL Database

A simple bash mysql is used to pipe the contents of the dumped SQL database back into a new mysql database.

$ mysql -u [username] -p [database] < [dump_file.sql]

Embedding Images in MP3 Files: version 0.0.3

This article has been superseded by a more recent posting.

This is the third revision of a python script for embedding artwork into MP3 (and other) audio files. Previous editions of this script only worked on the Linux platform. With the third revision, the script now uses the Mutagen python module and can be run on all major platforms:

  • Linux
  • Mac OS X
  • Windows
   1
   2
   3
   4
   5
   6
   7
   8
   9
  10
  11
  12
  13
  14
  15
  16
  17
  18
  19
  20
  21
  22
  23
  24
  25
  26
  27
  28
  29
  30
  31
  32
  33
  34
  35
  36
  37
  38
  39
  40
  41
  42
  43
  44
  45
  46
  47
  48
  49
  50
  51
  52
  53
  54
  55
  56
  57
  58
  59
  60
  61
  62
  63
  64
  65
  66
  67
  68
  69
  70
  71
  72
  73
  74
  75
  76
  77
  78
  79
  80
  81
  82
  83
  84
  85
  86
  87
  88
  89
  90
  91
  92
  93
  94
  95
  96
  97
  98
  99
 100
 101
 102
 103
 104
 105
 106
 107
 108
 109
 110
 111
 112
 113
 114
 115
 116
 117
 118
 119
 120
 121
 122
 123
 124
 125
 126
 127
 128
 129
 130
 131
 132
 133
 134
 135
 136
 137
 138
 139
 140
 141
 142
 143
 144
 145
 146
 147
 148
 149
 150
 151
 152
 153
 154
 155
 156
 157
 158
 159
 160
 161
 162
 163
 164
 165
 166
 167
 168
 169
 170
 171
 172
 173
 174
 175
 176
 177
 178
 179
 180
 181
 182
 183
 184
 185
 186
 187
 188
 189
 190
 191
 192
 193
 194
 195
 196
 197
 198
 199
 200
 201
 202
 203
 204
 205
 206
 207
 208
 209
 210
 211
 212
 213
 214
 215
 216
 217
 218
 219
 220
 221
 222
 223
 224
 225
 226
 227
 228
 229
 230
 231
 232
 233
 234
 235
 236
 237
 238
 239
 240
 241
 242
 243
 244
 245
 246
 247
 248
 249
 250
 251
 252
 253
 254
 255
 256
 257
 258
 259
 260
 261
 262
 263
 264
 265
 266
 267
 268
 269
 270
 271
 272
 273
 274
 275
 276
 277
 278
 279
 280
 281
 282
 283
 284
 285
 286
 287
 288
 289
 290
 291
 292
 293
 294
 295
 296
 297
 298
 299
 300
 301
 302
 303
 304
 305
 306
 307
 308
 309
 310
 311
 312
 313
 314
 315
 316
 317
 318
 319
 320
 321
 322
 323
 324
 325
 326
 327
 328
 329
 330
 331
 332
 333
 334
 335
 336
 337
 338
 339
 340
 341
 342
 343
 344
 345
 346
 347
 348
 349
 350
 351
 352
 353
 354
 355
 356
 357
 358
 359
 360
 361
 362
 363
 364
 365
 366
 367
 368
 369
 370
 371
 372
 373
 374
 375
 376
 377
 378
 379
 380
 381
 382
 383
 384
 385
 386
 387
 388
 389
 390
 391
 392
 393
 394
 395
 396
 397
 398
 399
 400
 401
 402
 403
 404
 405
 406
 407
 # * coding: utf-8 *
 import getopt, os, platform, subprocess, sys

 version  = "0.0.3"
 imageExt = [[".jpg", "image/jpeg"], [".png", "image/png"], [".gif", "image/gif"]]
 USE_MUTAGEN = "false"
 USE_FIRST_IMAGE_FOUND = "false"
 MUTAGEN_MODULE = ""
 CMD_CONVERT = "convert"
 CMD_EYED3 = "eyeD3"
 CMD_ID3 = "id3v2"
 DEBUG_FLAG = "false"
 DEFAULT_RESOLUTION = "600×600"
 coverJpg = "cover-embed-" + DEFAULT_RESOLUTION + ".jpg"
 coverPng = "cover-embed-" + DEFAULT_RESOLUTION + ".png"

 #######################################################################
 #
 # Method called to walk the directory tree
 #
 # args: extention to find
 # dirname: name of current directory
 # list of files in the current directory
 #
 def find(arg, dirname, names):
   foundMp3 = "false"
   for item in names:
     pathname = os.path.join(dirname, item)
     pathname = os.path.normpath(pathname)
     if os.path.isfile(pathname) and pathname.lower().find(arg)>0:
       debug("Found " + arg + " in " + dirname)
       foundMp3 = "true"

   if foundMp3 == "false":
     debug("No "+ arg +"s found in directory " + dirname)
     return

   # (1) Get an image to embed
   fullCoverJpg = findImage(dirname, names)

   if fullCoverJpg is None or len(fullCoverJpg) == 0:
     debug("No acceptable image found for " + dirname)
     return;
   
   # 2) Loop through contents and determine
   # if it is a file or a directory. Directories
   # can be skipped. Whereas for files, we'll
   # want to make a callback to embed the images.
   badAudioFiles = []
   for item in names:
     pathname = os.getcwd()
     pathname = os.path.join(pathname, dirname)
     pathname = os.path.join(pathname, item)
     pathname = os.path.normpath(pathname)
     if os.path.isfile(pathname) and pathname.lower().find(arg)>0:
       embedReturnCode = embedImage(pathname, fullCoverJpg)
       if 0 < embedReturnCode:
         badAudioFiles.append(pathname)
   if 0 < len(badAudioFiles):
     print "Errors occurred embedding images in the following files:"
     sys.stdout.write(" * ")
     print "\n\r * ".join(badAudioFiles)

 #######################################################################
 #
 # Given a directory, attempt to find a suitable cover image
 #
 def findImage(directory, filenames):
   imageFileToUse = ""
   # 1) Look in directory for if there is an
   # album cover, leave if not
   if coverJpg in filenames:
     debug("Found JPEG cover image, no conversion necessary")
     fullCoverJpg = os.path.join(directory, coverJpg)
     fullCoverJpg = os.path.normpath(fullCoverJpg)
     imageFileToUse = [fullCoverJpg, "image/jpeg"]
   elif coverPng in filenames:
     # No JPEG image found, but a legally named PNG exists
     imageFileToUse = convertImage(directory, coverPng, coverJpg)
   else:
     desiredImage = guessImageFile(directory, filenames);
     # found an acceptable image (based on extention)
     # if it is already a JPG, just make a copy. Otherwise do a
     # do a conversion
     if desiredImage  "":
       print >> sys.stderr, "no image found in " + directory
     elif USE_FIRST_IMAGE_FOUND  "true":
       imageFileToUse = desiredImage
     elif len(desiredImage[0]) > 0:
       imageFileToUse = convertImage(directory, desiredImage[0], coverJpg)
     # by virtue of making it out of the loop, we have admitted that
     # no acceptable image was present.
   return imageFileToUse;

 #######################################################################
 #
 # Simple heuristic for determining the correct cover art file.
 #
 def guessImageFile(directory, filenames):
   # There is no well-defined image for this directory.
   # Attempt to guestimate a reasonable replacement.  Look
   # through what is available & sort those 
   potentialImages = []
   for aFile in filenames:
     for extention,mime in imageExt:
       if aFile.lower().rfind(extention)  len(aFile)-len(extention):
 	debug("Looking at using file: " + aFile + " | Mime Type: " + mime)
 	fullpath = os.getcwd()
         fullpath = os.path.join(fullpath, directory)
         fullpath = os.path.join(fullpath, aFile)
         fullpath = os.path.normpath(fullpath)
         potentialImages.append([fullpath, mime])
       
   desiredImage = ""
   for currentFile, currentMime in potentialImages:
     if len(desiredImage) > 0:
       break;
     goodKeywords = [ "front", "large", "big", "cover"]
     for keyword in goodKeywords:
       if currentFile.lower().find(keyword) > 0:
         desiredImage = [currentFile, currentMime]
         debug("found image: " + currentFile)
         break;
 	
   # The first heuristic did not work, just pick the first image
   if desiredImage is None or desiredImage  "":
     if len(potentialImages) > 0:
       desiredImage = potentialImages[0];

   return desiredImage

 #######################################################################
 #
 # Linux command-line approach for converting/shrinking an image file.
 #
 # directory: directory where the file lives
 # foundImage: fullpath to the the desired image
 # desiredFilename: name of the output file
 #
 def convertImage(directory, foundImage, desiredFilename):
   # With imagemagick version 6.4.8.3, the command
   # to convert the image is similar to:
   # convert cover.png -geometry 600×600 cover.jpg
   fullCoverJpg = os.path.join(directory, desiredFilename)
   fullCoverJpg = os.path.normpath(fullCoverJpg)
   cmd = [CMD_CONVERT, fullCoverPng, "-geometry", DEFAULT_RESOLUTION , fullCoverJpg]
   shellCommandWrapper(cmd)
   return [fullCoverJpg, "image/jpeg"]

 #######################################################################
 #
 # Wrapper function to pick the correct means of embedding the 
 # album artwork into the audio file.
 #
 def embedImage(audiofile, image):
   addReturnCode = 0
   
   if USE_MUTAGEN:
     addReturnCode = embedImageViaMutagen(audiofile, image)
   else:
     addReturnCode = embedImageViaLinuxCommandLine(audiofile, image)

   return addReturnCode

 #######################################################################
 #
 # Use the "v2" approach of Mutagen
 #
 def embedImageViaMutagen(audiofile, image):
   returnCode = 0
   exec "from mutagen.mp3 import MP3"
   exec "from mutagen.id3 import ID3, APIC, error"
   mp3Object = MP3(audiofile, ID3=ID3)
   try:
     debug("Adding " + image[0] + " to the file " + audiofile )
     mp3Object.tags.add(APIC(encoding=3, mime=image[1], type=3, desc=u'Cover',data=open(image[0]).read()))
     mp3Object.save()
   except error:
     returnCode = 1

   return returnCode

 #######################################################################
 #
 # Use the "v1" approach of several command-line tools
 #
 def embedImageViaLinuxCommandLine(audiofile, image):
   removeReturnCode = 0
   addReturnCode = 0
   removeReturnCode  += zeroBpm(audiofile);

   # (1) Remove all existing images
   removeReturnCode += removeOtherImage(audiofile);
   removeReturnCode += removeOtherImage(audiofile);
   removeReturnCode += removeFrontImage(audiofile);
   removeReturnCode += removeFrontImage(audiofile);

   # (2) remove cruft
   removeReturnCode += removeCruft(audiofile);

   # (3) check if valid ID3 tag
   removeReturnCode += checkValidId3Tag(audiofile);

   # (4) add front image
   addReturnCode += addFrontImage(audiofile, image[0]);
   
   if 0 < removeReturnCode:
     debug("——————————————————————————————————")
     debug("Errors occured when removing/cleaning ID3 tag for: ")
     debug(audiofile)
     debug("This behavior is unexpected, but may not impact embedding the image.")
     debug("——————————————————————————————————")
     
   if 0 < addReturnCode:
     debug("——————————————————————————————————")
     debug("The unable to embed image in: ")
     debug(audiofile)
     debug("——————————————————————————————————")
   
   return addReturnCode

 #######################################################################
 #
 # Linux command-line wrapper for adding the front cover artwork
 #
 def addFrontImage(theFile, imageFile):
   imagePathArg = "—add-image=" + imageFile + ":FRONT_COVER" 
   cmd = [CMD_EYED3, "—no-color", imagePathArg, theFile]
   return shellCommandWrapper(cmd)

 #######################################################################
 #
 # Linux command-line wrapper for converting to the right ID3 version
 #
 def checkValidId3Tag(theFile):
   cmd = [CMD_EYED3, "—no-color", "—to-v2.4", theFile]
   return shellCommandWrapper(cmd)

 #######################################################################
 #
 # Linux command-line wrapper hack for dealing with floating-point BPMs
 #
 def zeroBpm(theFile):
   cmd = [CMD_EYED3, "—bpm=90", theFile]
   return shellCommandWrapper(cmd)

 #######################################################################
 #
 # Linux command-line wrapper for removing the front cover artwork
 #
 def removeFrontImage(theFile):
   cmd = [CMD_EYED3, "—no-color", "—add-image=:FRONT_COVER", theFile]
   return shellCommandWrapper(cmd)

 #######################################################################
 #
 # Linux command-line wrapper for removing the other artwork
 #
 def removeOtherImage(theFile):
   cmd = [CMD_EYED3, "—no-color", "—add-image=:OTHER", theFile]
   return shellCommandWrapper(cmd)

 #######################################################################
 #
 # Linux command-line wrapper for removing cruft
 #  * Mainly needed for older ID3 editors, may no longer be an issue
 #    on newer (2010+) installs
 #
 def removeCruft(theFile):
   cmd = [CMD_ID3, "-APIC", "", theFile]
   return shellCommandWrapper(cmd)

 #######################################################################
 #
 # Simple environment sanity check
 #
 def checkEnvironment():
   #
   # try to dynamically load mutagen
   #
   try:
     global MUTAGEN_MODULE 
     MUTAGEN_MODULE = import("mutagen")
     debug(MUTAGEN_MODULE)
     USE_MUTAGEN = "true"
     debug("Able to dynamically load mutagen")
   except:
     debug("Unable to dynamically load mutagen")
     if "Linux" <> platform.system():
       print >> sys.stderr, "*********************************************************"
       print >> sys.stderr, "***            Environment Check Failed!              ***"
       print >> sys.stderr, "***                                                   ***"
       print >> sys.stderr, "               mutagen must be installed"
       print >> sys.stderr, "***                                                   ***"
       print >> sys.stderr, "            http://code.google.com/p/mutagen/"
       print >> sys.stderr, "***                                                   ***"
       print >> sys.stderr, "*********************************************************"
       print
       sys.exit(1)
     #
     # fallback 'raw' method, look for the appropriate command-line utilities
     #
     checkEnvironmentHelper([CMD_EYED3, "—help"])
     checkEnvironmentHelper([CMD_ID3])
   if "Linux"  platform.system():
     checkEnvironmentHelper([CMD_CONVERT])
   else:
     global USE_FIRST_IMAGE_FOUND
     USE_FIRST_IMAGE_FOUND = "true"
   
 #######################################################################
 #
 # Helper method for displaying errors about some simple command-line
 # environment checks (for Linux)
 #
 def checkEnvironmentHelper(command):  
   if 0 < shellCommandWrapper(command):
     print >> sys.stderr, "*********************************************************"
     print >> sys.stderr, "***            Environment Check Failed!              ***"
     print >> sys.stderr, " Unable to locate '" + command[0] + "' in the PATH"
     print >> sys.stderr, "*********************************************************"
     print
  
 #######################################################################
 #
 # Prints debugging messages (if the debugging flag has been set)
 #
 def debug(message):
   if DEBUG_FLAG  "true":
     print message

 #######################################################################
 #
 # Small wrapper method for executing Linux Shell Commands
 #
 def shellCommandWrapper(command):
   FNULL = open(os.devnull, 'w')
   process = subprocess.Popen(command, shell=False, bufsize=1, \
             stdin=None, stdout=FNULL, stderr=FNULL)
   process.wait()
   return process.returncode
   
 #######################################################################
 #
 # Print Usage Information
 #
 def usage():
   print
   print "This script takes one or more arguments that are expected to be a "     + \
         "a single directory or set of directories. For each argument supplied, " + \
         "the script will recurse into the directory and look for an image file. If a " + \
         "file is found, it will be converted to the default resolution (" + DEFAULT_RESOLUTION + \
         "). The script will then look for any MP3s in the directory. If found, the " + \
         "image will be embedded into each file (overwriting an existing embedded image)."

 #######################################################################
 #
 #
 # Start of the 'main()' function
 # 
 def main():
   #######################################################################
   #
   # Read Command-Line
   #
   try:
     opts, args = getopt.getopt(sys.argv[1:], "hd", ["help", "debug", \
                                "use-first-image", "use-mutagen"])
   except getopt.GetoptError, err:
     # print help information and exit:
     print >> sys.stderr, err
     usage()
     sys.exit(2)
   #######################################################################
   #
   # Parse Command-Line Options and Arguments
   # 
   for o, a in opts:
     if o in ("h", "-help"):
       usage()
       sys.exit()
     elif o in ("d", "-debug"):
       global DEBUG_FLAG
       DEBUG_FLAG = "true"
     elif o  "--use-first-image":
       global USE_FIRST_IMAGE_FOUND
       USE_FIRST_IMAGE_FOUND = "true"
     elif o  "—use-mutagen":
       global USE_MUTAGEN
       USE_MUTAGEN = "true"
     else:
       assert False, "unhandled option"
   #######################################################################
   #
   # Now that all the processing is out of the way, walk the directory
   # structure and embed artwork into any discovered MP3 files.
   #
   if len(args) == 0:
     print >> sys.stderr, "Missing command-line arguments"
     usage()
   else:
     checkEnvironment()
     for directory in args:
       os.path.walk(directory, find, ".mp3")

 if name == "main":
     main()
 

Download the Script:

Highlighting Code Syntax with Pygmentize Redux

Here is a quick update for the Pygmentize/textpattern script that fixes the handling of ‘+’ characters.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 #!/bin/bash

 for var in "$@"
 do
   # Generate HTML
   pygmentize -O colorful,linenos=1,full,classprefix=code_ \
    -o $var.1.html $var

   # Add space in front of each line
   sed 's/^/ /' $var.1.html > $var.2.html
   
   # Replace + with &#43; to prevent textpattern markup from
   # reinterpreting as internal markup that leaving hanging
   # /span> tags
   sed 's/\+/\&#43;/g' $var.2.html > $var.html

   rm -f $var.1.html $var.2.html
 done