[Tfug] MySql and rsync

[TR]

On 4/17/07, Stephen Hooper <stephen.hooper at gmail.com> wrote:
>
> On 4/17/07, Terence Rudkin <trudkin at gmail.com> wrote:
> > Any MySQL experts out there?
> > My company has a program using rsync to load files into a mysql
> database.
> > It appears to mostly work, but some files are not getting inserted.
> The
> > process is loading about 5 records a second / 16k hour, not sure how
> big  a
> > record is.
> > Anyone with expertise is using rsync with mysql?   Or even Mysql tuning
> >
>
> Hi Terence,
>
> Can you be more specific about what types of files you are rsyncing?
> Are they flat files?  Files containing formatted data (like in a CSV
> form)?
>
> >From your description, I am imagining files that are being saved in
> one place (say JPG's from a web application), and then rsynced to a
> server with a MySQL database where something comes by, picks them up,
> and does an SQL insert.  Am I close?


Yes  close byt it is an  rsync  command that is  doing the work.    When I
first posted I did no have a lot of background on this.  But I know now that
image files are sent to a working directory when a modified version of rsync
sends the file to the mysql db.   It is based on the
patches/ODBC-dblog.diff.  This adds logging to the rsync daemon.  This
was further modified to do
an  insert into the database. for our specific needs.   By counts the number
staged to be send is greater then those showing up in the database.

How about files with a bunch of outgoing phone calls separated by
> tabs, and dumped by a PBX every night, then rsynced to a server with a
> MySQL database where something comes by, picks them up, reads them in,
> breaks out the appropriate fields, and makes a bunch of SQL inserts.
> Am I closer with that one?
>
> Anyways, I would say that whatever answer you get depends a lot on the
> types of files... (but, in any case I am not a MySQL expert :) ).
>
> _______________________________________________
> Tucson Free Unix Group - tfug at tfug.org
> Subscription Options:
> http://www.tfug.org/mailman/listinfo/tfug_tfug.org
>