Akismet.com
Dear Spam-Comment and Spam-Track-Back authors,
Say hello to my little friend, Akismet.
Along with the corresponding WordPress plugin, I feel we’ll be seeing less of you around here. Especially nice that your comments will be auto-deleted if I ignore them for a month.
Have a great life; don’t work too hard.
Corwin
I’ll expand on this sometime soon, over on my Perl blog. Meanwhile, here’s the code.
This bit of MySQL (three tables, two functions and a stored procedure) plus a perl script will create a catalog of all of the files on one or several drives attached (or mapped) to a win32 system. I’m running it using ActivePerl under Windows XP. Not sure yet if it’s compatible with more interesting versions of perl for windows (e.g. strawberry). It tries to identify duplicated files a provide an instance count for them. Unique is considered to distinct values for name (ignoring drive/path), MD5 ‘fingerprint’, or file-size. So, if you have two version of a file with exactly the same name and content in two different folders, that’s a single instance. On the other hand, if you have two versions of the same file with different names those are considered distinct, even if the size and content are identical between them.
assuming active perl, use it with something like
arc.pl c:\
or
arc.pl c:\ d:\
or
arc.pl c:\some\folder c:\some\other\folder
#!perl
use strict;
use warnings;
use Cwd q(abs_path);
#use File::Basename q(basename);
use File::Find q(find);
use DBD::mysql;
use Digest::MD5::File qw(file_md5);
use POSIX q(strftime);
use Win32::DriveInfo q(VolumeInfo);
use File::Spec;
our $DEBUG = 1;
my ($db,$user,$pass) = ('db_name','db_user','db_pass');
my $dbh=DBI->connect(qq(dbi:mysql:database=$db;host=some.database.server.com),$user,$pass);
my $sql = q(SELECT get_instance_count(?,?,?,?,?,?,?,?,?) AS count);
my $sth = $dbh->prepare($sql) or die;
my @dirs = !@ARGV ? abs_path() : map abs_path($_), @ARGV;
search( $_ ) for @dirs;
exit( 0 );
sub search {
my $folder = shift;
unless (-d $folder) {
warn qq(SKIPPING: missing or not a folder: $folder\n);
return;
}
my ($volume,$directories,$file) = File::Spec->splitpath( $folder );
my @volume_info = Win32::DriveInfo::VolumeInfo($volume);
my $volume_sth = $dbh->prepare(q(select get_volume(?,?,?) AS volume_id)) or die;
$volume_sth->execute( @volume_info[0,1,3]) or die;
my $volume_id = $volume_sth->fetchrow_arrayref->[0];
find(sub{ store($volume_id, $File::Find::name) unless -d $File::Find::name }, $folder);
}
sub format_time {
my $format = '%Y-%m-%d %H:%M:%S';
strftime($format, gmtime( shift ));
}
sub store {
my $volume = shift;
my $fullname = shift;
unless (-r $fullname) {
warn qq(SKIPPING: missing or unreadable: $fullname\n);
return;
}
my $md5 = file_md5( $fullname );
unless (defined $md5 and $md5) {
warn qq(failed to generate MD5 for $fullname\n);
return;
}
my (undef,$directories,$basename) = File::Spec->splitpath( $fullname );
#my $basename = basename( $fullname );
$fullname = File::Spec->catfile( $directories, $basename );
my($extension) = $basename =~ /\.([^.]+)$/ ? $1 : '';
my @stat_parts = stat $fullname;
my($size,$atime,$mtime,$ctime) = @stat_parts[7..10];
$sth->execute( $volume,
$fullname, $basename, lc( $extension ),
format_time($atime), format_time($ctime), format_time($mtime),
$size, $md5) or die;
my $refcount = $sth->fetchrow_arrayref->[0];
$DEBUG && printf "%5d %s\n",$refcount,$fullname;
return $refcount;
}
And here’s the SQL:
/**
* DDL script for the ARC database (mysql)
*/
-- DROP DATABASE ARC_LIVE;
-- CREATE DATABASE ARC_LIVE;
-- USING ARC_LIVE;
DROP FUNCTION IF EXISTS get_instance_count;
DROP FUNCTION IF EXISTS get_volume;
DROP PROCEDURE IF EXISTS get_canon;
DROP TABLE IF EXISTS volume;
DROP TABLE IF EXISTS canon;
DROP TABLE IF EXISTS instance;
CREATE TABLE volume (
id int(11) NOT NULL AUTO_INCREMENT,
name varchar(255) NULL,
serial binary(16) NOT NULL,
filesystem varchar(255) NULL,
added_dttm timestamp NOT NULL,
PRIMARY KEY( id ),
UNIQUE KEY `volume_UNIQUE_INDEX` (serial) USING HASH
) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8;
CREATE TABLE canon (
id int(11) NOT NULL AUTO_INCREMENT,
bytesize bigint unsigned NOT NULL,
md5 binary(16) NOT NULL,
basename varchar(255) NOT NULL,
extension varchar(255) NULL,
refcount int(11) unsigned NOT NULL,
PRIMARY KEY( id ),
UNIQUE KEY `cannon_UNIQUE_INDEX` (basename,md5,bytesize) USING HASH
) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8;
CREATE TABLE instance (
id int(11) NOT NULL AUTO_INCREMENT,
volume int(11) NOT NULL REFERENCES volume(id),
canon int(11) NOT NULL REFERENCES canon(id),
fullname varchar(32767) NOT NULL,
atime timestamp NOT NULL,
ctime timestamp NOT NULL,
mtime timestamp NOT NULL,
added_dttm timestamp NOT NULL,
PRIMARY KEY( id )
) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8;
DELIMITER $$
CREATE FUNCTION get_volume(
in_name varchar(255),
in_serial binary(16),
in_filesystem varchar(255)
)
RETURNS INT
NOT DETERMINISTIC
BEGIN
DECLARE lc_volume int(11);
SELECT id INTO lc_volume FROM volume WHERE serial = in_serial;
IF lc_volume IS NULL THEN
INSERT INTO volume( name, serial, filesystem, added_dttm )
VALUES( in_name,in_serial,in_filesystem, NOW() );
SELECT LAST_INSERT_ID() INTO lc_volume;
END IF;
return lc_volume;
END $$
CREATE PROCEDURE get_canon(
IN in_basename varchar(255),
IN in_extension varchar(255),
IN in_bytesize bigint unsigned,
IN in_md5 binary(16),
OUT canon_id int(11)
)
BEGIN
DECLARE lc_refcount int(11);
SELECT id, refcount
INTO canon_id,lc_refcount
FROM canon
WHERE basename = in_basename
AND md5 = in_md5
AND bytesize = in_bytesize;
IF ( canon_id IS NULL ) THEN
INSERT INTO canon( bytesize, md5, basename, extension, refcount )
VALUES( in_bytesize,in_md5,in_basename,in_extension, 1 );
SELECT LAST_INSERT_ID() INTO canon_id;
ELSE
UPDATE canon
SET refcount = lc_refcount+1
WHERE id = canon_id;
END IF;
END $$
CREATE FUNCTION get_instance_count(
in_volume int(11),
in_fullname varchar(32767),
in_basename varchar(255),
in_extension varchar(255),
in_atime timestamp,
in_ctime timestamp,
in_mtime timestamp,
in_bytesize bigint unsigned,
in_md5 binary(16)
)
RETURNS INT
NOT DETERMINISTIC
BEGIN
DECLARE lc_refcount int(11);
DECLARE lc_canon int(11);
SELECT canon INTO lc_canon
FROM instance
WHERE fullname = in_fullname
AND volume = in_volume;
IF lc_canon IS NOT NULL THEN
return 0;
END IF;
CALL get_canon( in_basename,in_extension, in_bytesize, in_md5,
lc_canon );
INSERT INTO instance( volume, canon, fullname, atime, ctime, mtime,added_dttm )
VALUES( in_volume,lc_canon,in_fullname,in_atime,in_ctime,in_mtime,NOW() );
SELECT refcount INTO lc_refcount FROM canon WHERE id = lc_canon;
return lc_refcount;
END $$
DELIMITER ;
Closer and closer we go- Each daring step a gambit, Each timid gambit a dare. Held sacred in a moment, Lost phrases tightly tacit, Closer and closer we go. Whither our song my student of reckless wants illicit? Each timid gambit a dare? Whose lesson most imprudent- To wear the tear implicit, Closer and closer we go! Loose our madness abient? Shelter thread and grommet? Each timid gambit a dare? A torrid tangle rampant Wizened knots insipid. Closer and closer we go Each timid gambit a dare.
(c)2011 Corwin Brust, CC BY-ND
DON’T PANIC
Probably nothing.
First, having one’s account cracked into, stolen, or similar is a pretty common occurrence and not something to worry over much about. (See: Bruce Schneier’s interview on MRP’s All Things Considered on April 5th, 2011. Also, since MPR didn’t see fit to provide one, here’s a link to Crypto-Gram, Bruce’s monthly security newsletter. If you are into security and related technology issues and don’t get this, do.)
Next, it is very unlikely to be your fault, at least in any useful sense. Yes, if you are surfing around porn sites and not using Firefox “Private Browsing” or similar, you are exposing yourself to unnecessary risk but, to paraphrase Bruce from the aforementioned interview, your alternative to putting your personal information out in the world is to live in a cave. Realistically, you are going to submit information to people you (quite rightfully) trust, and at some point that information is going to fuel an attack, against your email account for example. Given enough attacks, your account will eventually be hacked and you will send your friends some pretty embarrassing stuff. Get over it.
Finally, it really is unlikely that you did anything wrong. Yes, it is possible you have clicked on the wrong thing from Facebook or Twitter or girls-who-like-boys-who-live-with-parents.com, but it is just as likely your account info was pulled from a file or database somewhere –most likely one that really should have been using the Google API instead of storing your password. In any event, while the malware may be painful it’s not going to be the end of your electronic universe, and it’s certainly not going to keep you from getting back to reading emails.
Speaking to G-mail, specifically, the process for getting back into control of your account is quite simple.
If you are still able to access your account, just change your password and set about letting your friends know your gmail account had been comprimised and to ignore (and avoid follow links within) any emails from you during the period of compromised. (This is the prescription once you regain access also.)
Very often, once an attacker has successfully cracked into your account he will change your password. Fixing this is still relatively trivial. On the gmail sign in page you will see a link, right below the “Sign In” button that will guide you though the process:
You will be prompted for your gmail email address.
Next you’ll be challenged by what’s called a CAPTCHA, meant to keep computer programs (so called “robots” or “bots”) from using the same page for Evil™. In practice these can be a bit of a pain, especially if you are trying to do this from a mobile device or just have a small or low-resolution screen. Here are a couple of tricks I’ve found to help:
Control + R (PC) or Option + R (Mac) may do the trick, or you can try pressing the “Back” button and re-submit. You can do this several times until you get one that’s readable on your device. Each time you reload the page the web-browser will likely confirm with you that it is okay to “Repost Form Data”. This is just going to re-submit your entry from the previous page (i.e. your gmail email address) and that’s just fine.
] It takes a good ear and you may have to listen a few times. Note that this is not reading you the same code as from the image; it is giving you a different code composed only of numbers so anything you hear that isn’t a number is part of the noise added to confuse potential non-human consumers. You can also use trick #1 (reload) in combination if there’s a particular digit you just can’t make-out.Once you get past the CAPTCHA you will be given several options to complete the recovery process.
If you’ve setup a mobile number with Google that’s the easiest way. Google will text you with a short numeric code and provide a prompt box. When you get the text, just enter the code and you’ll be able to change the password for your gmail/Google account. If your account has not yet been compromised, and you have a mobile phone which you can use to receive text messages, go register your mobile number with Google right now. I have yet to get a text message from Google as a result of doing this except in testing for this post. If you are logged into Google from the browser from which you are reading this, one or the other of these links should take your directly to the correct settings page where you can add (or edit) your mobile number.
If you provided an email address to Google when you setup your gmail account, and that account still exsits, this is your next best bet. When you setup a new gmail account you usually need to supply a different -already existing- email address. If that account is still functional -hasn’t also been hacked, didn’t belong to a old ISP, etc- and you can still access it, use this. This password reset method is just about as quick as the SMS method and equally painless. Usually the top option, you will see bits and parts of your other email address to be used, with some characters replaced with asterisks (*), to protect your privacy — bit late for that, eh? Just make sure the corresponding radio button is selected, press continue, wait for the email to arrive over on your other account, click on the link it will contain and you are good to go. To save time, make sure you really can access this other account before you press “Continue”.
If you didn’t have a mobile setup, and whatever previous email you may have had associated with your gmail/Google account isn’t or isn’t accessible anymore, you will have to take the long road. Google has attracted millions of users to dozens of products on the basis of making the hard way pretty darned simple, so don’t worry – you’re in good hands.
You should, if at all possible, complete this from a computer and physical location from which you have previously had success using your gmail. You’ll be asked to fill in a form. When you submit that form Google will take note of the IP address (read: numerical internet address) from which you are connected at the time of submission. Google also attempts to track and maintain a list of ‘known’ addresses for your account with exactly this purpose in mind, so if most of your gmail logins are occurring from one place (your home, for example) that’s where you should be when you do this. Also, you may thinking “I have broadband at home and I’m DHCP”. While it’s true that it’s not a guarantee that you will have the exact same IP address for any previous connection it’s likely to be in the same subnet or at least class-C (read: general neighborhood). Moreover, the process of getting you back in charge of your gmail account will ultimately be performed by a person.
You will need to select “I no longer have access to these” and then press “Continue”. On the following page you are asked for a pile of information. Generally you want to complete as much as you can as accurately as you can.
As stated before, the decision to trust you and give you back your account will fall to a person, who will review records, consult with peers and ultimately make a decision. Fundamentally, your goal is going to be to make the decision to give you back your account as easy as possible.
First, the Googler helping you will need a means of contacting you, specifically an email address. (Ironic? Maybe. They don’t generally seem to like to talk on the phone unless there’s money on the table, I find.) You can use a spouse’s or a work email (assuming this would not be some sort of breach) but it may make the most sense just to hop over to Hotmail (where last I checked you don’t need a working email address to create a new account) and create a new ‘throw-away’ account. You can just this just for communication with Google. Later on, when you’re back in the saddle with your gmail, you can add this into the Google Account Recovery Settings page linked above.
With the question of how you will communicate with Google while your situation is under review out of the way, you are onto the easy stuff. Assuming you have been sending spam to your friends and contacts you probably want to select “I believe someone has taken over my account”. In any case, answer the rest of the questions to the best of your knowledge and ability. To the date-related questions, get as close as you can. It’s well understood by technologists that people’s memories are often fuzzy in this area.
As a final hint, the last question on the form asks for the last password you knew for your account. You don’t want to mistype this; I suggest you enter this into a Notepad or Text Wrangler or whatever plain text editor you like (emacs) and then paste into the form field to avoid such trivial errors.
Good luck and let me know how it works out for you!
A useful version of my first twitter app is complete. Registered to twitter as ”Steven Brust Quotes”, you can access it here: http://bru.st/skzb/QuoteOfTheDay/
Now that I’ve finished a few more features (like preventing the same quote from showing up by chance day after day), I’m ready to call this working. Go ahead and let me know how you’ve broken it. I wrote in detail about the initial version on my perl blog. As mentioned, I’m pleased with how this went and I’ve really enjoyed getting my feet wet with the Perl API for twitter. With the new features, and including HTML the code-base is up to 275 lines. (Note, differing from the version I shared on my Perl blog, line counts no longer include the data, as that’s been moved into a database.)
Here’s the feature list:
I have no specific plans to extend this list of features; here’s the short list of things I on which I probably won’t work:
If you would really like to see any of this you should let me know. Thanks in advance for your feedback. Enjoy!
Corwin
[sig omitted]
As was reported today on Business Insider, IBM’s stock value has surpassed that of Microsoft. When I first learned of this on twitter, I was initially inclined to be quippy. Upon further reflection, this is a very good sign.
From the perspective of someone who uses products from both companies as part of making my living in technology, IBM’s value surpassing that of Microsoft is a very good sign. None of this is to say that Microsoft is a bad company and IBM is a good one – both are fine companies with fantastic stuff I love to work with. What this is, is a closer reflection to actual value reflection, comparing them. To understand why, we need to look a bit a both companies.
Microsoft started as a software firm, building a company out of the demand for an operating system for the brand new personal computing market. IBM started out in the hard-ware market making typewriters and office equipment. While both companies have expanded well beyond their origins neither has escape them in terms of public perception: we continue to think of Microsoft as a software company, and IBM as an hardware company. More significant is the path each company has taken to industry dominance. Microsoft owes its current fortunes entirely to one product – the Windows operating system. While many of it’s other offering have been successful -and are indeed very cool- it’s story is mostly about turning dominance in the personal computer operating system market into additional opportunity. Conversely, IBMs rise is owed to the diversity of it’s product line: even while marking the very hardware which catapulted Microsoft to the top, they remained the key manufacturer of main-frames, type-writers and other key business machines.
In other words, while Microsoft was amassing wealth in an emerging market (small-scale computing), IBM was carefully adding new products to service this market into it’s existing model of servicing the equipment needs of businesses of all scales. The important point here is that IBM has always been a cautious company able to deliver reliable product to markets as they emerge. In contrast, Microsoft is more of a boom-rider attempting to ride each new wave of buzz which can be made to service it’s entrepreneurial feed-back loop.
Finally, as business continues to be a key market segment for both companies, take notice that while most corporations continue to show preference to the Microsoft business software tool-chain (i.e. Windows, ActiveDirectory, Exchange) these same companies tend to show preference also to IBM, and in terms of hardware as well as software (M-queue series and Mercury Quality Center as well as the Blade and P-series servers.) This is critical because it shows that while both IBM and Microsoft will likely continue loosing market in the business computing segment to Apple, Red Hat, Google and non-commercial offerings, Microsoft is considerably more vulnerable because it is less diversified within the segment. (If a company goes
Out of all this, I take heart in seeing IBM edge past Microsoft in terms of public market valuation. This suggests to me the bubble around the technology sector may have deflated. Notwithstanding the inevitable speculation, that we are seeing to prices based on valuation. That’s best for everybody.
Here’s a short Perl program that will spit out an HTML page showing you color swatches. Change the list of colors after the “__DATA__” (end-of-program-start-of-data) line in the source below. Tested under Windows XP using ActivePerl 5.10.1 but it should work just about anywhere.
I finally got decent code-formatting going, you can still help yourself to
Comments welcomed.
#!/usr/local/bin/perl
use strict;
use warnings;
my @colors = grep defined && length,
map /(#[a-z0-9]{3,6})/ig,
split /\s+/ms,
<DATA>;
print <<'END_HEAD';
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Color Swatches</title>
<style>
div { width:100%; height:25px }
span { width:4em; display:block; float:left }
</style>
</head><body>
END_HEAD
for my $color (@colors)
{
print qq( <div style="background-color:$color">\n);
print qq( <span style="color:$_">$_</span>\n) for @colors;
print qq( </div>\n);
}
print <<END_FOOT;
</body>
</html>
END_FOOT
__DATA__
#000
#026227
#414958
#4E5869
#6F7D94
#5D9732
#8090AB
#DDD
#FFF
Greetings and thanks for checking-in!
I’ve got quite a bit on my plate just now, so updates to this site are slow in coming.
If you are looking for project development or technical support for a project with kick-off in the second half of 2011 now’s the time to reach out! Get the most rapid response by writing directly to MetaVerse Project Support (commercial) or to me (non-Commercial & barter).
Thanks for checking-in – catch ya on the fun way!