Showing posts with label TooTechnical. Show all posts
Showing posts with label TooTechnical. Show all posts

Sunday, 28 June 2015

Fighting with markets (and software)

Despite being moderatly quiet on the blog, I am still busy in Eve.

My current project that I don't understand anywhere near well enough is how to use the Crest API, for pulling market information.

What I used to do.


I had previously used google docs and eve central's API, but google docs are flaky on something as small as my PI spreadsheet, and is unworkable for a full market listing.

What I am currently doing.


I use Steve's python based crest downloader to pull down all buy and sell orders for a region ( https://github.com/fuzzysteve/CREST-Market-Downloader ). I have some success, but what works reliably for him in pulling down an entire region of market orders in 15 minutes, takes 4 hours for me and only works 1/3 or the time.

I also pull down my current asset list and market orders using jeveassets, peeking into both corp and personal assets.  Despite having my own division in the corps that I am currently in, I have to filter out 'others' assets from my own. (using sql per below)

While I am no wizard at either Perl or Java, I am very good at SQL.  So I upload these 2 data sets and put them into a MariaDB instance (MySQL fork/clone - but much less hassle to download)

Knowing how to map item ids to names, groups, categories and meta levels is useful.  For that I use Steve's SDE dump, and as MariaDB/MySQL is my preferred database, https://www.fuzzwork.co.uk/dump/mysql-latest.tar.bz2 works well (other databases and versions in the same folder).

With all of this, a final SQL script can show me, by whatever market group I want to filter on, what my current stock counts are, and what is currently on market, the local price (if any), the jita price, my markup.

When this comes together, it works well.  When it doesn't it is a pain.

Alternatives

Evernus is a reasonable alternative for some of this functionality and I have looked at it.  It is not enough.

I cant work out how to make it show me all the skillbooks missing off the local market, then swap to look at either T1 or T2 ammo, then flit to the next part of the market I am interested in.  I have  2 tycoon and 1 wholesale level pilots, but there are 400 different skillbooks alone, and I have less than 700 market slots. If I was better at working out what was needed, I could probably make sure that was there, but have little chance of stocking everything wanted.

I got mildly excited by seeing Evernus Alliance Market ( https://bitbucket.org/krojew/evernus-alliance-market ).  It looks to be a good group logistics tool, and despite being written for a Linux environment, with a bit of fiddling, got it installed on a windows instance (there are a couple of evernus 'bug reports' with tips that I found useful ).  But .. it too didnt have the functionality I needed, much of which is in Evernus (including a quick crest sales/buy orders download).

The future

I am (mostly) enjoying the challenges of doing nullsec trading, but I want better tools.  Evernus is an open source software package, and I am in the process of downloading it plus the tools required to enhance it.

Friday, 1 May 2015

Re-install of Eve

As you might gather from the title, I had issues ... with my Eve install.

I run multiple concurrent sessions of eve, described in more detail in http://foo-eve.blogspot.com.au/2014/06/running-multiple-eve-launchers-my.html

Compared to many of you, I also have a poor internet connection, so I also run with downloading with all data.

The first symptom was the Eve launcher warning me that I had problems with the cache of the skins file.

The second symptom was that after that Eve would not start.  The task manager (I run windows 8) showed there was something running, but nothing would happen.  And I could not even get the above error message.

I tried using the task manager to crash Eve to restart.
I tried rebooting my computer ( always try this ).
I tried evefile.exe
I tried different install folders

All of these failed.  Sigh.

Ok delete the Program Files (x86)\CCP folder and reinstall.  Still failed .... Grr.

Delete the local cache folder ... and restart ... Nope

Delete %localappdata%\CCP\EVE and restart ... Success.  I can log in.

Download on demand makes the initial Eve experience much faster. 

I am still not sure about everything.  I tried to bring up a PI management window and can now have a largely unusable session with a black screen, apart from what is behind the overview window.  I can't bring up the options window (for a safe logoff).

I  logged a Help desk ticket (being able to paste but not write makes this frustrating but *shrug*, whenever you are lodging help desk tickets, it doesn't take much.

But ... for those who sent me emails the last couple of days, this is why I have been slow to respond




Saturday, 7 March 2015

Google docs whack-a-mole continued, plus a request of readers

My whack-a-mole with google docs is continuing.



I have added caching and a random delay to my open copy of the trading spreadsheet.  https://docs.google.com/spreadsheets/d/1qL_ofk-QRQCFt2zKTTEXSi6oY4n1gtZX6Mojxxtgs-4/edit#gid=1869404277

I think it should be ... less unstable than it was before.

Readers, can you please have open (and leave open) the above spreadsheet for a few days, using either Firefox or Chrome.  Eve's in game browser has no chance.  Internet explorer and Opera struggle.  It is a published document and I don't mind if you keep it open longer.

A full refresh (daily) will take a couple of minutes to load.  Apart from that, it should be fairly quick.

If it is breaking, please leave a comment on this page.  If you find yourselves using it (either the original - which I can track, or a copy - which I can't), please also leave comments about how.

This is a continuation of http://foo-eve.blogspot.com.au/2015/03/market-spreadsheets-in-space.html


Thursday, 5 March 2015

Whack-a-mole with Google Docs and Eve Central

My primary PI spreadsheet ( https://docs.google.com/spreadsheets/d/1lXM8PqU0Tn4EJJ2xRuI7uo55IrZ7MKWJidK9T_bA9L4/edit#gid=0 ) is fixed again.


In my last post about Google Docs, I inadvertently tripped a google doc limitation.




I have added caching to this spreadsheet, and temporarily disabled it in my other spreadsheets.  This should not only resolve my urlfetch issue, but also significantly speed up the loading of the spreadsheet.  It comes at a 'cost' of encouraging the data to be up to 6 hours older.

I will be rolling the changes out to the other spreadsheets soon.

I also found a github code snippet from 'Haggen', also putting together google docs based eve-central calls https://gist.github.com/haggen/461a454b200efe89f3e6

For those interested, I have raised a bug on Fuzzy Steve's github repository, https://github.com/fuzzysteve/eve-googledocs-script/issues/5

Sunday, 1 February 2015

Techical update on Google docs & Eve Central

Google docs and the Eve central API for data has not been playing nicely.

I wrote up my implementation of a solution based on Steve Ronuken's google script fix, that works moderately well.

https://docs.google.com/spreadsheets/d/1lXM8PqU0Tn4EJJ2xRuI7uo55IrZ7MKWJidK9T_bA9L4/edit#gid=0

http://foo-eve.blogspot.com.au/2014/12/pi-spreadsheet-working-again.html

The problem I had is that Google was caching data for far too long, and I missed some price changing trends.

The solution is to use a ?cachebuster=<value> parameter when you want a new call.

From the Eve Central API, apparently we are allowed to pull the same data every 5 minutes.  ( https://groups.google.com/forum/#!topic/eve-central/b5FjssiqFIQ ).

For my purposes, I don't care about completely current pricing, and price refreshing makes my spreadsheet unusable for about 30 seconds while it happens.  My current solution is to pass the day of the month as the cachebuster parameter, that is I request different data once per 24 hours.

I will be keeping a bit more of a closer eye on this as PI prices have moved around a bit over the last few weeks (post scheduled for Tuesday my time).

Thursday, 8 January 2015

Packaged vs Unpackaged volumes

This is just a minor post about some spreadsheets about space work I have been doing in Eve, primarily regarding volume. If you are not involved in spreadsheets in space (either as spreadsheets or applications), please move along as there will be nothing to see.

In Eve, it is possible to have a 'packed' volume different to the 'unpacked' volume.  For most items these are the same.  For ships and containers at least, they are different.

After perusing the Eve static data, web searching, and finally asking Steve Ronuken, it appears that while the unpacked volumes are available in the static data, there is no available source of packed volumes apart from the Eve software itself.

There is a very old forum post with partially incorrect values.

So I have created a new google doc to share with my investigation : https://docs.google.com/spreadsheets/d/1hlng1FppLB40oACK76LpG9r-qZZORrFgqRTXognMlDw/edit#gid=0

This contains 4 tabs:
  • Volumes by type id and type name for containers.  It appears that there is some groupings according to container size but even then there are exceptions
  • Volumes by group id and group name for ships.  It appears that there are groupings according to ship class (frigate through super capital). 
  • MySQL compatible sql to create tables with this data and then join it back to the eve static data
  • Attributions
It is entirely possible that I have gotten items incorrect, and/or missed classes of items.  Eventually this data will become stale.

Regardless, I am hauling to null (not always personally), and this has attached a cost to it, I want to know if it is worth hauling completed items or hauling component parts.  Outside highsec, each trip's risk and cost are based on number of trips more than what is inside a hauler.   Of course, one day, one of our haulers will make someone a glorious killmail.


Monday, 23 June 2014

Running multiple Eve Launchers - my solution

This post is way to technical, not quite fit for player consumption.  It is however fit for developer consumption, and I am publish this in the hopes that someone with a Windows scripting background runs with it.  If you do not have a scripting background, your eyes are liable to glaze over.

Once upon a very very long time ago, on a computer before the ones we are using now; when we wanted to run multiple concurrent eve clients, we made junctions of our Eve folders.  Each time we wanted to change a pilot, we had to enter a user name and password.

Once upon a time, but more recently than that, when we wanted to run multiple concurrent clients we called the Eve exe once per concurrent client.  Each time we wanted to change a pilot, we still had to enter the username and password.

Today, here and now, when you want to change pilots inside an account, we have this 'wonderful' Eve launcher that allows us to remember our accounts user name and password and access the Eve store, but ...  you must re-enter the username/password when you want to change pilots between accounts.

I find this frustrating; especially when resetting my PI.  Planetary Interaction is limited by the number of planets you have.  More alts with PI means more planets.  But using alts means more logging in.

My old method of resetting PI is :
  • Log onto every account, selecting the pilot that has been offline the longest.
  • Do PI.  At any time there is a break move onto the next account's pilot; Especially during the warp from POCO back to POS
  • When finished with a pilot log onto the next longest offline pilot on that account.
  • Whine about having to re-enter username/password when swapping accounts.  Sometimes get up and for some cheese.
When you do this on ... too many accounts it becomes tiresome.  There are some PI pilots I know with way more accounts than I run.  The same issues may apply for heavy research, manufacturing and market accounts.

An Eve folder contains the following files and folders
  • A launcher folder
  • Other folders include bin, bulkdata, lib, res and script
  • A whole series of files, including 
    • *.stuff files, containing textures, sounds and other in game resources
    • *.patch files, apparently can be deleted once applied
    • *.exe files, being the actual executables
    • *.ini files, containing various some configuration information
    • cacert.pem and ca-bundle.crt which I believe to do with encryption
    • debug.log, obvious file is obvious
    • code.ccp, manifest.dat, __access_check__; file names are indicative, but *shrug*

So I googled; and found lots of entries from before the 'new' launcher. 

I tried to copy the launcher folder itself and running the launcher.exe that is in it; no luck, it would not start a new launcher window.

What roughly worked was copying the entire eve folder.  This allowed me to have 2 Eve launchers concurrently, but copying 18GB of data takes a long time as well as a fair amount of disk space.  It also means that your system can't cache the same files; meaning lower overall system performance.  I wanted better.

What I am now using.


4 Launcher windows all open at the same time. 
Apart from cropping and obscuring the account name, these accounts are all active at the same time.

Each additional Eve instance takes 174MB
Again, cropping and obscuring personal information.

I have a script 'makeclone.sh' that takes an Eve folder, and replicates it
  • Copying the launcher folder as is, where is;  This obviously is copying stuff not needed (including old log and browser cache files), but is quick and small enough
  • For all other folders, junctions into the replicated folder back to the originals in the primary Eve folder.  These must be junctions, symbolic links did not work.
  • All files are hard linked into the replicated folder back to the original files in the primary Eve folder
All of this takes about 15 seconds per replicated folder,  at 1% of a full Eve installation.  This could be made smaller by working out what is needed out of the Eve installation; I suspect that a launcher/cache folder structure without files would save space, but so far have not been bothered adding that complexity to my script.  It also is very flexible in coping with future CCP changes. 
  • Update the 'master' normally with  the launcher.
  • re-run the script to blow away the existing replicated folders, and make new versions of them.

On my version of windows, the personal settings are in %localappdata%\CCP\EVE, with a new fresh set of files for every replicated folder.  I am contemplating changing my script to copy my primary installation's settings over to the replicated folders, but have not yet decided.  I like the idea of updating my overview once, and then automatically cloning it, but have not gotten that far yet.


My command prompt environment and scripting language, is a bash shell from cygwin ( https://www.cygwin.com/ ).  The script itself is available as a pastebin or google drive document.

As always, too many words, not enough pretty pictures. I also suspect that for the majority of readers, this takes the prize for 'Driest Foo post ever'.   On top of that, I have based my scripts on my skillset; which is a Unix/Linux based scripting language in a Windows environment; so it is not yet even useful to the majority of you, at least not yet.  It is also likely useless for Mac users.

What I do know is that there are others with more Windows based scripting knowledge than I have, that could easily translate this into a Dos batch file, or failing that a Powershell script, so it is available to the wider Eve community. 

Assuming you have actually read this far, if you have something like this script that anyone can use, either based on this work or independently created, please leave a comment.  Similarly, if you are lacking in scripting skills and want something like this without the cygwin dependancy, also leave a comment.