Link: https://goo.gl/forms/HTYt4UVPB2TffteW2
I'm working on a project related to card searching, so I'm conducting this anonymous survey to help me understand how people search for cards and what they search for. It's a very quick survey (I think), and any answers are extremely helpful and appreciated. Thanks
I'm getting a bug whenever it attempts to download data for a card starting with AE before the great ligature update.
System.InvalidCastException: Conversion from type 'DBNull' to type 'String' is not valid.
at Microsoft.VisualBasic.CompilerServices.Conversions.ToString(Object Value)
at Gatherer_Extractor.FenetrePrincipale.Treatment_MAJ(Int32 nbre_tache, BackgroundWorker worker, DoWorkEventArgs e)
at Gatherer_Extractor.FenetrePrincipale.BWkMAJ_DoWork(Object sender, DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)
This is with the latest version, 4.1c, on Windows 10.
Yeah, gatherer's inaccuracies have been a constant source of pain for the mtgjson folks; I hang out in their Gitter channel and a ton of the Kamigawa flip cards had their names mangled recently. And just for fun, Gatherer also deleted the Vintage, Legacy and Commander as formats. They just keep making mistake after mistake, somehow...
For image extraction, have you tried Gatherer Extractor? It's what I used to get all my MTG images for phash recognition. Its only drawback is not getting multiple images of cards like Icatian Javelineers and the basic lands.
I think hardcast_sixdrop has the best idea of what to do for the mtg-specific optimizations, since he made them for the Torch7 network. I briefly considered attempting to implement them in Tensorflow, but I have about 3 separate-but-related projects going on right now, so I don't have time to squeeze this one in too. If you wanna take it on, it won't be easy, but I assume hardcast would be helping you out in that.
It looks like the folder you chose for the -data_dir, data/custom_encoding, doesn't contain a file called input.txt. You may have an input file there, but unless it's called input.txt it won't work (it's picky like that).
What you're looking for is hardcast's tutorial (which also includes GPU instructions). Alternatively, if you want to stick to Windows, you can use Python 3.5 and TensorFlow as described in my earlier post. That should get you going; if you have any questions though we should be able to fix (most of) them
That... seems like a hell of a script. I wouldn't even know where to start with something like that, but I've never worked with advanced python IO, so there's that. What kind of interface would the rnn library need to support this?
In the meantime, I ran the TensorFlow char-rnn code with 3 layers and rnn_size of 512, and I got some interesting results, such as...
shahki, skirch flock 3R
legendary creature ~ elder dragon (rare)
flying
when @ enters the battlefield, if you control three or more artifacts, return target creature card from your graveyard to the battlefield.
at the beginning of your upkeep, if @ is in your graveyard, you may pay {BBR.}. if you do, return @ from your graveyard to the battlefield.
(5/5)
A perfectly legible and sensible legend! Apart from the . in the mana cost. Unfortunately there seems to be some overfitting because one of the cards had the name 'krark~clan shaman' and the other completely copied the text of Demonic Torment. Not sure how to turn up/down the temperature of the new TF sampling script yet. Luckily the mtg-encode script works fine with the output though, which is very convenient. Here's a few more:
long~term intellect 5UU
creature ~ avatar (rare)
trample
when @ enters the battlefield, draw three cards.
(11/11)
// An 11/11 trampling for 7, with upside? Yes, please. In blue no less. At least it's rare!
shambling shieldmage 3B
instant (common)
cast @ only during the declare attackers step and only if you've been attacked this step.
prevent all damage that would be dealt to you this turn. if it's blocking, remove a % counter from it. if that card is returned this way, you may pay 1. if you do, you gain 2 life.
// Impossible instructions are impossible. How has one been attacked if this can be cast only during the declare attackers step?
grixisk strike R
creature ~ elemental (common)
echo 3R
@ attacks each turn if able. R: @ gains indestructible until end of turn.
sacrifice @: destroy target enchantment.
(2/2)
// I'm afraid this gives red a really good way to deal with enchantments, so that wouldn't fly in today's colour pie world.
lifelink 2RR
instant (rare)
strive ~ @ costs 1U more to cast for each target beyond the first.
return any number of target creatures you control. prevent all combat damage and exile them. then that player shuffles his or her library.
// An actual correct implementation of Strive! That's pretty impressive. Pity the second line is a bit garbled.
destructive retribution 3G
enchantment (rare)
when @ enters the battlefield, search your library for a creature card with power 2 or less, reveal that card, and players.
// REVEAL THE PLAYERS!
veteran war cry 3
creature ~ snake elf and (rare)
hexproof
@ gets +1/+1 as long as you control a permanent other than @.
whenever @ deals combat damage to a player, if @ is in your graveyard, creatures you control get +1/+1 until end of turn.
(3/3)
// Ah, the rare 'and' subtype. I wonder if it's possible to deal combat damage from the graveyard, technically? If yes, yay anthem!
press to dust 2
artifact (rare)
sunburst T, remove X % counters from @: target creature gets -X/-X until end of turn.
countertype % echois
// Technically... this works? Sunburst implies the number of 'echois' counters (if we assume they're charge counters of a sort), and it's a keyword ability so it technically doesn't need reminder text! And this is a rare example of X being both defined and used. Winner!
I'd forgotten how fun it is to generate random cards. There'll probably be a few more of these posts.
The new library is much more powerful. Instead of reading a text file, it can invoke a linux program that streams training text over a variable number of channels. This is nice because you can write whatever custom, dynamic training regimen you want
What does this do exactly? Opening a file is pretty simple conceptually; you open a book and read the pages. This streaming over channels is more like having the individual pages shot at you all at once. What advantages, concretely, does it convey over just opening the .txt file of the corpus and splitting it up based on our categories? When you say 'custom, dynamic training regimen', I don't see why that can't be applied to a properly parsed .txt corpus file.
Also, if it's a Linux program, it might not work for Windows TF, which would suck. My goal (not sure if it's achievable, but one can hope) is to have all mtg-rnn stuff in Python so it can work with Windows or Linux, alongside TF. I'm feeling adventurous in that I want to write probably 3 different neural nets in TF (all doing different things with card data; images, syntax and this generation one), so learning tricks in this one couldn't hurt. It feels like an insane goal, given I've never written a neural net from scratch and I have barely enough comp-sci/maths background, but heck. Learning!
I was afraid they would add something new like that. The github way to add stuff to my repo is to submit a pull request. If it's just a minor change, it might not be worth the effort. I'll have time to look at it later this week.
It's trivial; basically wherever you define other symbols, you just add definitions for C and E. Not really worth making a pull request over.
Yep, work's still ongoing. Talcos just showed me this paper; imagine its results applied to MTG art.
Does your code support new stuff like {C} and {E}? I know my local copy of mtg-encode does, but I wasn't able to do a thing with git to apply the changes to your repo.
I also have my own usable but only partially complete and totally undocumented library for doing character-level language model things, which is faster and more flexible than mtg-rnn. I might have a chance to finish that as well over the holidays. It should make it really easy to do things like curriculum learning, but unfortunately it is not Windows friendly. I wonder how hard it would be to port the API over to TensorFlow...
What is this new library you speak of? Is it more effective at generation than mtg-rnn? Porting it to TF would be great in the sense that the audience who could use it would increase, since people with Ubuntu systems can operate as normal, but now Windows people could do it too.
Speaking of porting, what changes exactly did you make to mtg-rnn as compared to char-rnn? I'm thinking about making those same changes to the TF char-rnn, but I'm not exactly sure what to do with it yet.
There's actually already something kinda like that. The only part of the corpus that needs more encoding is the rules text (since you can't compress colour, cost, type, p/t, any more than what it is now) and I'm working on figuring out a 'programmatic' representation of it which would be useful for training, but also for other search-related things.
I finished training the default corpus in the TF char-rnn, and I got some results!
|5enchantment|4|6|7|8|9whenever a creature, and put it on top of his or her library into his or her graveyard.|3{^^}|0N|1wand hood|
|5sorcery|4|6|7|8|9prevent all colorless {UU} costs {^RR} gain life and you no island into his or her library this creature. its controller reveals his or her hand, attacking creatures.\-&^^^^^^: you get an equipment by name is red.|3{^^^UUUU}|0A|1boros gatekeeper|
OK, so they're not great results, but it's vaguely coherent... Boros Gatekeeper is a really, really good name, considering Boros is on Ravnica, which has gates, and... yeah, I'm surprised at that. And it proves the theory, at least. Now, the upgrading can begin.
Hey folks! It's been a while, but I (with a huge amount of help from Talcos) have been working on some pretty interesting stuff involving cards and neural nets. No results yet, but one thing I can say is that image recognition networks have a hell of a hard time distinguishing between Angels and Birds in card art.
I remember a while back there was a lot of discussion about making the char-rnn stuff run on Windows. For ages, the 'best' option was simply to take a VM and run Ubuntu in that, but that prevents GPU usage. I myself set up a dual-boot to be able to use GPUs in Ubuntu natively. Now, however, Google has blessed us with TensorFlow for Windows, complete with GPU support! My installation experience was pretty trivial. Just download the official Python 3.5 installer, run it, and do the steps in the TF docs. It's one step, really, and make sure you install the GPU version.
As a test, beyond the basic validation test they have you run, I was lucky enough to find a char-rnn implementation of Karpathy's algorithm in TF. I just cloned the repository, went in, and ran the train.py script on the default Shakespeare training corpus. It runs 23100 batches in 17 minutes on my GTX 980ti, which isn't bad (I haven't tried the equivalent on my Ubuntu install, but I think 0.04s per batch was about the same as I was getting there). After the training finished, I ran the sample.py script and it generated Shakespearean text, so it all seems to work great. It'll probably need a few tweaks for making Magic cards, but the best part is it's all in Python, so the output should be handled by hardcast_sixdrop's existing scripts so you can still generate html/MSE2 sets out of this.
One thing about Windows; it obviously has a lot more stuff running in the background. This isn't really an issue for the CPU, since it's a GPU-intensive task, but one thing that happens is that a lot more GPU memory is used in Windows than Ubuntu. I suspect that turning down some graphics settings to free up some of that memory would help (for context, TF detected I had 4.9 of 6GB free to be used, while on Ubuntu, Chainer (another ML framework library) has access to 5.6GB).
Anyway, hopefully this is useful to those who wanna run this stuff on Windows. I'll try to get actual Magic card generation working to see what modifications might have to be made, hopefully they won't be too extensive. But I'm just really glad that proper, GPU-accelerated machine learning is possible on Windows now.
edit: Doing a test run with the standard text input now, I suspect sampling it will return something just about acceptable. It's running at 51300 instead of 23100, so it'll take about 40 mins to train. hardcast_sixdrop did quite a few modifications to the original char-rnn code, mostly involving loading the data and randomizing it better and such, which are going to be challenging to replicate in TensorFlow, since, well, it's a separate library to Torch, which is what we've used so far. But if I can't do it, I'm sure someone else can.
I see the id-name-# feature in new release, thanks
Another request; right now downloading a full set's scans only downloads one of each basic land. Could you make it so that it downloads all the basic lands (perhaps labeled e.g. Island (273))?
I like his playing of 4 Jaces. Chaining them together to remove/play threats from the opponent's deck is really cool. Also the Tamiyo + Gideon combo is neat. Might have to try his list on XMage.
I'm working on a project related to card searching, so I'm conducting this anonymous survey to help me understand how people search for cards and what they search for. It's a very quick survey (I think), and any answers are extremely helpful and appreciated. Thanks
This is with the latest version, 4.1c, on Windows 10.
For image extraction, have you tried Gatherer Extractor? It's what I used to get all my MTG images for phash recognition. Its only drawback is not getting multiple images of cards like Icatian Javelineers and the basic lands.
For additional info, check the original RNN repo by Karpathy and his subsequent talk video on them. Since he wrote the char-rnn network, he's rather good at explaining them.
In the meantime, I ran the TensorFlow char-rnn code with 3 layers and rnn_size of 512, and I got some interesting results, such as...
shahki, skirch flock 3R
legendary creature ~ elder dragon (rare)
flying
when @ enters the battlefield, if you control three or more artifacts, return target creature card from your graveyard to the battlefield.
at the beginning of your upkeep, if @ is in your graveyard, you may pay {BBR.}. if you do, return @ from your graveyard to the battlefield.
(5/5)
A perfectly legible and sensible legend! Apart from the . in the mana cost. Unfortunately there seems to be some overfitting because one of the cards had the name 'krark~clan shaman' and the other completely copied the text of Demonic Torment. Not sure how to turn up/down the temperature of the new TF sampling script yet. Luckily the mtg-encode script works fine with the output though, which is very convenient. Here's a few more:
long~term intellect 5UU
creature ~ avatar (rare)
trample
when @ enters the battlefield, draw three cards.
(11/11)
// An 11/11 trampling for 7, with upside? Yes, please. In blue no less. At least it's rare!
shambling shieldmage 3B
instant (common)
cast @ only during the declare attackers step and only if you've been attacked this step.
prevent all damage that would be dealt to you this turn. if it's blocking, remove a % counter from it. if that card is returned this way, you may pay 1. if you do, you gain 2 life.
// Impossible instructions are impossible. How has one been attacked if this can be cast only during the declare attackers step?
grixisk strike R
creature ~ elemental (common)
echo 3R
@ attacks each turn if able.
R: @ gains indestructible until end of turn.
sacrifice @: destroy target enchantment.
(2/2)
// I'm afraid this gives red a really good way to deal with enchantments, so that wouldn't fly in today's colour pie world.
lifelink 2RR
instant (rare)
strive ~ @ costs 1U more to cast for each target beyond the first.
return any number of target creatures you control. prevent all combat damage and exile them. then that player shuffles his or her library.
// An actual correct implementation of Strive! That's pretty impressive. Pity the second line is a bit garbled.
destructive retribution 3G
enchantment (rare)
when @ enters the battlefield, search your library for a creature card with power 2 or less, reveal that card, and players.
// REVEAL THE PLAYERS!
veteran war cry 3
creature ~ snake elf and (rare)
hexproof
@ gets +1/+1 as long as you control a permanent other than @.
whenever @ deals combat damage to a player, if @ is in your graveyard, creatures you control get +1/+1 until end of turn.
(3/3)
// Ah, the rare 'and' subtype. I wonder if it's possible to deal combat damage from the graveyard, technically? If yes, yay anthem!
press to dust 2
artifact (rare)
sunburst
T, remove X % counters from @: target creature gets -X/-X until end of turn.
countertype % echois
// Technically... this works? Sunburst implies the number of 'echois' counters (if we assume they're charge counters of a sort), and it's a keyword ability so it technically doesn't need reminder text! And this is a rare example of X being both defined and used. Winner!
I'd forgotten how fun it is to generate random cards. There'll probably be a few more of these posts.
Also, if it's a Linux program, it might not work for Windows TF, which would suck. My goal (not sure if it's achievable, but one can hope) is to have all mtg-rnn stuff in Python so it can work with Windows or Linux, alongside TF. I'm feeling adventurous in that I want to write probably 3 different neural nets in TF (all doing different things with card data; images, syntax and this generation one), so learning tricks in this one couldn't hurt. It feels like an insane goal, given I've never written a neural net from scratch and I have barely enough comp-sci/maths background, but heck. Learning!
It's trivial; basically wherever you define other symbols, you just add definitions for C and E. Not really worth making a pull request over.
Does your code support new stuff like {C} and {E}? I know my local copy of mtg-encode does, but I wasn't able to do a thing with git to apply the changes to your repo.
What is this new library you speak of? Is it more effective at generation than mtg-rnn? Porting it to TF would be great in the sense that the audience who could use it would increase, since people with Ubuntu systems can operate as normal, but now Windows people could do it too.
Speaking of porting, what changes exactly did you make to mtg-rnn as compared to char-rnn? I'm thinking about making those same changes to the TF char-rnn, but I'm not exactly sure what to do with it yet.
I finished training the default corpus in the TF char-rnn, and I got some results!
|5enchantment|4|6|7|8|9whenever a creature, and put it on top of his or her library into his or her graveyard.|3{^^}|0N|1wand hood|
|5sorcery|4|6|7|8|9prevent all colorless {UU} costs {^RR} gain life and you no island into his or her library this creature. its controller reveals his or her hand, attacking creatures.\-&^^^^^^: you get an equipment by name is red.|3{^^^UUUU}|0A|1boros gatekeeper|
OK, so they're not great results, but it's vaguely coherent... Boros Gatekeeper is a really, really good name, considering Boros is on Ravnica, which has gates, and... yeah, I'm surprised at that. And it proves the theory, at least. Now, the upgrading can begin.
I remember a while back there was a lot of discussion about making the char-rnn stuff run on Windows. For ages, the 'best' option was simply to take a VM and run Ubuntu in that, but that prevents GPU usage. I myself set up a dual-boot to be able to use GPUs in Ubuntu natively. Now, however, Google has blessed us with TensorFlow for Windows, complete with GPU support! My installation experience was pretty trivial. Just download the official Python 3.5 installer, run it, and do the steps in the TF docs. It's one step, really, and make sure you install the GPU version.
As a test, beyond the basic validation test they have you run, I was lucky enough to find a char-rnn implementation of Karpathy's algorithm in TF. I just cloned the repository, went in, and ran the train.py script on the default Shakespeare training corpus. It runs 23100 batches in 17 minutes on my GTX 980ti, which isn't bad (I haven't tried the equivalent on my Ubuntu install, but I think 0.04s per batch was about the same as I was getting there). After the training finished, I ran the sample.py script and it generated Shakespearean text, so it all seems to work great. It'll probably need a few tweaks for making Magic cards, but the best part is it's all in Python, so the output should be handled by hardcast_sixdrop's existing scripts so you can still generate html/MSE2 sets out of this.
One thing about Windows; it obviously has a lot more stuff running in the background. This isn't really an issue for the CPU, since it's a GPU-intensive task, but one thing that happens is that a lot more GPU memory is used in Windows than Ubuntu. I suspect that turning down some graphics settings to free up some of that memory would help (for context, TF detected I had 4.9 of 6GB free to be used, while on Ubuntu, Chainer (another ML framework library) has access to 5.6GB).
Anyway, hopefully this is useful to those who wanna run this stuff on Windows. I'll try to get actual Magic card generation working to see what modifications might have to be made, hopefully they won't be too extensive. But I'm just really glad that proper, GPU-accelerated machine learning is possible on Windows now.
edit: Doing a test run with the standard text input now, I suspect sampling it will return something just about acceptable. It's running at 51300 instead of 23100, so it'll take about 40 mins to train. hardcast_sixdrop did quite a few modifications to the original char-rnn code, mostly involving loading the data and randomizing it better and such, which are going to be challenging to replicate in TensorFlow, since, well, it's a separate library to Torch, which is what we've used so far. But if I can't do it, I'm sure someone else can.
Another request; right now downloading a full set's scans only downloads one of each basic land. Could you make it so that it downloads all the basic lands (perhaps labeled e.g. Island (273))?