Solve system of equations with complex coefficients and conversions to polar angle form.

Is it complex or matrix? You can have only one mode at a time.What kind of equations? The solve function uses real-valued equations.
Complex numbers: You can add, subtract, multiply and divide. You can convert a number from rectangular to polar forms and vice-versa; you can calculate the absolute value of a complex number, or a conjugate.
All the operations are done in the Mode CMPLX.

Your scientific calculator is unable to solve complex equation with complex coefficients. You should try to solve by hand directly using the quadratic formula or by factoring the polynomial in z
failing that, another way would be to set z=x+iy, substitute this for z, carry out the algebra and try to separate real and imaginary parts. But your two equations will constitute a system of two quadratic equations. I am not aware of any general method to solve coupled nonlinear equations.
Good luck.

Thhe Casio FX-9860G SD can solve a polynomial equation of degree 2 or 3 with REAL coefficients. If the complex MODE is set to REAL it will find the real roots. If the complex mode is set to a+ib, it will find the real and complex roots.

Apparently it will take coefficients that are real, and will give a Ma Error if any coefficient is complex.

Addendum.
The calculator CANNOT solve equations with complex coefficient. YOU can however convert the system of linear equations with ccomplex coefficients ( of the type you show) as a system of 4 linear equations in 4 unknowns; Split x into a real and an imaginary part, split y into a real and an imaginary part. Substitute Real(x)+iIm(x) for variable x in the equations; substitute Real(y)+iIm(y) for y in the two equations; do the algebra. In each of the original equations split the Real and imaginary parts. You should be able to derive 4 linear equations in unknowns Real(x), Im(x), Real(y), and Im(y).
Use the linear equation solver to obtain the solutions. Recompose x=Real(x)+iIm(x), and y=Real(y)+iIm(y)

Alternatively, after you create the system of 4 linear equations you can use the matrix utility to find Real(x), Im(x), Real(y) and Im(y) and recompose the x and y.

The Casio FX-9860G SD can solve a polynomial equation of degree 2 or 3 with REAL coefficients. If the complex MODE is set to REAL it will find the real roots. If the complex mode is set to a+ib, it will find the real and complex roots.

Apparently it will take coefficients that are real, and will give a Ma Error if any coefficient is complex.

Addendum.
The calculator CANNOT solve equations with complex coefficient. YOU can however convert the system of linear equations with ccomplex coefficients ( of the type you show) as a system of 4 linear equations in 4 unknowns; Split x into a real and an imaginary part, split y into a real and an imaginary part. Substitute Real(x)+iIm(x) for variable x in the equations; substitute Real(y)+iIm(y) for y in the two equations; do the algebra. In each of the original equations split the Real and imaginary parts. You should be able to derive 4 linear equations in unknowns Real(x), Im(x), Real(y), and Im(y).
Use the linear equation solver to obtain the solutions. Recompose x=Real(x)+iIm(x), and y=Real(y)+iIm(y)

You have presumably one variable (x) and a parameter k. This equations represents a family of quadratic curves. The only way you can solve for parameter k is if you give at least two coordinate points (x_1,y_1) and (x_2,y_20 through which the curve passes. Failing that you cannot.
I suggest you check your problem again and better still check the theory before attempting to solve any problems. Sorry.

Apply what you learned, especially that this system is quite simple.
Elimination
1. entails eliminating one variable to find (in this case) a single equation involving the other variable.
2. Solve that new equation, meaning isolate the variable that was not eliminated.
3. Substitute the value found in the last step and replace it in one of the original equation to obtain the other variable.

x+y=10
x-y=-6

Add the left sides to get x+y+x-y=2x
Add the right sides of the system to obtain 10+(-6)=4
Write Sum of left sides =Sum of right sides or 2x=4.
Since this new equation involves only x, you can solve it for x, getting x=4/2=2
Now you know the value of x (=2)
Take one of the original equations (for example x+y=10) and put 2 in place of x.
The equation becomes 2+y=10
Solve it to obtain y=10-2=8
Thus your solution is x=2, y=8

Check: use one equation in which you substitute 2 for x and 8 for y: x-y=-6 becomes 2-8=-6
Verified.
In the general case you will have to multiply by certain values to obtain opposite coefficients for the same variable. Here that was not necessary because the coefficient of y is 1 in the first equation and -1 in the second.

Select the EQN computational mode 5:EQN

For system of linear equations in 3 unknowns select 2: in the screen below.

Arrange the variables in your equations in the same order (x first, y second, and z third).
The coefficients to enter in the editor are the factors of the variables: In your case a_1=10, b_1=-3, c_1=10, d_1=5, a_2=8, b_2=-2, c_2=9, d_2=3; a_3=8, b_3=1, c_3=-10, d_3=7.
To enter d_1, d_2, and d_3 you will have to scroll to the right to reach the cells where they should go.

Once finished entering the coefficients, press EXE to get the solutions. You may have to use the arrow Down to display the y- and z-solutions.
I verified that the matrix is non-singular and the system has a solution. In fraction form
x=27/53, y=-1 and 22/53; finally z=-23/53

You have to put the equations into matrix form first. To do this, each variable has one column in the first matrix and you fill in the co-efficients for the variables. The second matrix has one column and contains all the numbers.

{ 2 -1 1 -1} = Matrix A
{ 1 3 -2 0}
{ 3 -2 0 4}
{-1 -3 -3 -1}

{-1} = Matrix B
{-5}
{ 1}
{-6}

{x=-2} = A*(B^-1)
{y=-.2}
{z=3}
{w=1/6}

I used excel for all my calculation and a helpful tutorial can be found here. I hope this helps and have a nice day!

I am afraid you cannot use the TI8xPlus family of calculators to solve linear systems in matrix form. In this calculator, matrices must have real coefficients.
You can however separate (expand) the problem into a linear system of 4 equations in 4 unknowns and try to solve it with the calculator.

1. define X = a+i*b
2. Define Y=c+i*d
3. Rewrite the first equation substituting a+i*b for X and c+i*d for y.
4. Gather all real terms together, and all imaginary terms together on the left.
5. You will have (8a-15b-8c) +i(15a+8b-8d) = 10 +0i
6. This equation can be split into two equations by saying that

• the real part pf the left member is equal to the real part of the right member, this gives 8a-15b-8c=10, and
• the imaginary part of the left member is equal to the imaginary part of the right member, this gives 15a+8b-8d=0

Similarly, you rewrite the second complex equation substituting a+ib for X and c+id for Y, then expand the binomial products, gather the real parts on the left together, and the imaginary parts on the left together. After that you equate the real part on the left to the real part on the right, and do the same for the imaginary parts.

If I did not make mistakes during the expansions and the gathering of terms you should get the following equation
-(8a+8c+10d) +i*(-8b+10c-8d) =0+i*0 from which you extract an equation for the real parts, -(8a+8c+10d)=0 and another for the imaginagy parts i*(-8b+10c-8d) =i*0

If I did not make mistakes (you should be able to find them, if any) your system of two linear equations with complex coefficents has been converted to a system of 4 linear equations with real coefficients.

8a-15b-8c=10
15a+8b-8d=0
8a+8c+10d=0
-8b+10c-8d =0

Now, you can in theory solve this system with help of the calculator, to find a, b, c, and d. When these are found, you can reconstruct the X and Y solutions.

Now get to work: Ascertain that my extracted equations are correct, then solve for a, b,c, and d, and reconstruct X and Y.
I am no seer, but my hunch is that this system is degenarate. I will not explain what that means.

I assume you are speaking of solving a system of equations with a number of unknowns. If not, please correct me. Here's an example in practice:

If you have a system of 3 equations with 3 unknowns, you would set up your matrix so that the coefficients of each variable for a particular equation are on one row. So, given equations x + y + z = 0, 2x + 3y - 4z = 1, x + -z = -1 you would type the following into your calculator: [[1,1,1,0][2,3,-4,1][1,0,-1,-1]] and press enter to make sure you typed it correctly. notice that in the third row there is a zero, since we have zero time y for the third equation. Then row-reduce the matrix (2nd > 5 > 4 > 4 or in the CATALOG as rref). You should get out the matrix [[1,0,0,-1][0,1,0,1][0,0,1,0]]. This says that x=-1 y = 1 z=0 since my first column contained the coefficients for the x variable, the second for the y variable, and the third for the z variable. The last column contains the solution, the part on the other side of the equals sign.

Hope this helps! For more reading (from someone else; I just made this one up), check out the Wikipedia articles on Gaussian elimination and Systems of linear equations